Image processing apparatus, image processing method, and computer program product

ABSTRACT

An image processing apparatus includes an imaging unit which is mounted on a mobile object and picks an image of a predetermined field of view to generate an image; and a moving information detection unit which detects moving information including a speed of the mobile object. The apparatus also includes a processing content setting unit which sets contents of a process to be performed in the image generated by the imaging unit, based on the moving information detected by the moving information detection unit; and a processing calculation unit which performs processing calculation according to the contents of the process set by the processing content setting unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of PCT international application Ser.No. PCT/JP2006/309419 filed May 10, 2006 which designates the UnitedStates, incorporated herein by reference, and which claims the benefitof priority from Japanese Patent Applications No. 2005-137850, filed May10, 2005; No. 2005-145825, filed May 18, 2005; and No. 2005-145826,filed May 18, 2005, and all incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, an imageprocessing method, and a computer program product which pick an image ofa predetermined field of view and perform image processing to agenerated image.

2. Description of the Related Art

A conventional inter-car distance detection apparatus which is mountedon a given vehicle such as an automobile, processes an image obtained bypicking a preceding vehicle running in front of the given vehicle, anddetects a distance from the given vehicle to the preceding vehicle areknown (For example, see Japanese Patent No. 2635246). In order tocapture the inter-vehicle distance detection apparatus captures thepreceding vehicle on an image, a plurality of distance measurementwindows are set at predetermined positions in an image, an image isprocessed in each of the set distance measurement windows to calculate adistance to an arbitrary object, and recognizes an image pickup positionof the preceding vehicle on the basis of the calculation result andposition information of the measurement windows.

A technique which, in order to detect a road surface condition of a roadin a traveling direction in a traveling state of a given vehicle, picksan image in front of the vehicle and recognizes a predetermined objecton the picked image is also known (for example, see Japanese Patent No.3290318). In this technique, by using the picked image, a driving lanedividing line such as a white line or a driving lane dividing zone suchas a central reservation on a road on which the given vehicle runs isrecognized.

SUMMARY OF THE INVENTION

An image processing apparatus according to an aspect of the presentinvention includes an imaging unit which is mounted on a mobile objectand picks an image of a predetermined field of view to generate animage; a moving information detection unit which detects movinginformation including a speed of the mobile object; a processing contentsetting unit which sets contents of a process to be performed in theimage generated by the imaging unit, based on the moving informationdetected by the moving information detection unit; and a processingcalculation unit which performs processing calculation according to thecontents of the process set by the processing content setting unit.

An image processing method according to another aspect of the presentinvention includes picking an image of a predetermined field of viewfrom a mobile object to generate an image; detecting moving informationincluding a speed of the mobile object; setting contents of a process tobe performed in the generated image, based on the detected movinginformation; and performing processing calculation according to thecontents of the process set.

A computer program product according to still another aspect of thepresent invention has a computer readable medium including programmedinstructions for performing image processing on an image generated by animaging unit which is mounted on a mobile object and picks an image of apredetermined field of view to generate the image. The instructions,when executed by a computer, cause the computer to perform detectingmoving information including a speed of the mobile object; settingcontents of a process to be performed in the generated image, based onthe detected moving information; and performing processing calculationaccording to the contents of the process set.

The above and other objects, features, advantages and technical andindustrial significance of this invention will be better understood byreading the following detailed description of presently preferredembodiments of the invention, when considered in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of an image processingapparatus according to a first embodiment of the present invention;

FIG. 2 is a flow chart showing procedures performed until the imageprocessing apparatus shown in FIG. 1 completely outputs distanceinformation;

FIG. 3 is an explanatory diagram conceptually showing distancecalculation based on image data obtained by picking image by a stereocamera;

FIG. 4 is an explanatory diagram showing correspondence between rightand left image regions before a rectification process;

FIG. 5 is an explanatory diagram showing correspondence between rightand left image regions after a rectification process;

FIG. 6 is a flow chart showing procedures of an arrival point predictionprocess;

FIG. 7 is a diagram for explaining a method of calculating an arrivalpoint in the arrival point prediction process;

FIG. 8 is a diagram showing an example of a result obtained when apredicted arrival point is overlapped on an image picked by an imagingunit;

FIG. 9 is a flow chart showing procedures of an calculation rangesetting process;

FIG. 10 is a diagram showing an example of a result obtained when ashort-distance window is overlapped on an image picked by the imagingunit;

FIG. 11 is a diagram showing an example of a result obtained when along-distance window is overlapped on an image picked by the imagingunit;

FIG. 12 is a diagram showing an example of a result obtained when amiddle-distance window is overlapped on an image picked by the imagingunit;

FIG. 13 is a block diagram showing a configuration of an imageprocessing apparatus according to a second embodiment of the presentinvention;

FIG. 14 is a flow chart showing procedures of an arrival pointprediction process;

FIG. 15 is a diagram for explaining a method of calculating an arrivalpoint in the arrival point prediction process;

FIG. 16 is a diagram for explaining a prediction function selected inthe arrival point prediction process;

FIG. 17 is a diagram showing an example of a result obtained when ashort-distance window is overlapped on an image picked by an imagingunit;

FIG. 18 is a diagram showing an example of a result obtained when along-distance window is overlapped on an image picked by the imagingunit;

FIG. 19 is a block diagram showing a configuration of an imageprocessing apparatus according to a third embodiment of the presentinvention;

FIG. 20 is a diagram for explaining a method of correcting an arrivalpoint in an arrival point prediction process;

FIG. 21 is a block diagram showing a configuration of an imageprocessing apparatus according to a fourth embodiment of the presentinvention;

FIG. 22 is a block diagram showing a configuration of an imageprocessing apparatus according to a fifth embodiment of the presentinvention;

FIG. 23 is a flow chart showing procedures performed when distanceinformation in the image processing apparatus shown in FIG. 22;

FIG. 24 is a flow chart showing procedures of a window switching processshown in FIG. 23;

FIG. 25 is a diagram showing an example of a low-speed window selectedby the window switching process in FIG. 24;

FIG. 26 is a diagram showing an example of a high-speed window selectedby the window switching process in FIG. 24;

FIG. 27 is a diagram showing a state obtained when a low-speed window isoverlapped on a picked image;

FIG. 28 is a diagram showing a state obtained when a high-speed windowis overlapped on a picked image;

FIG. 29 is a diagram showing an example of distance information formedby an calculation unit;

FIG. 30 is a diagram showing an example of distance information formedby the calculation unit;

FIG. 31 is a block diagram showing a configuration of an imageprocessing apparatus according to a sixth embodiment of the presentinvention;

FIG. 32 is a flow chart showing procedures performed until distanceinformation in the image processing apparatus shown in FIG. 31;

FIG. 33 is a flow chart showing procedure of a window switching processshown in FIG. 32;

FIG. 34 is a diagram showing an example of a right window selected bythe window switching process in FIG. 33;

FIG. 35 is a diagram showing an example of a left window selected by thewindow switching process in FIG. 33;

FIG. 36 is a diagram showing an example of a through-traffic windowselected by the window switching process in FIG. 33;

FIG. 37 is a block diagram showing a configuration of an imageprocessing apparatus according to a seventh embodiment of the presentinvention;

FIG. 38 is a flow chart showing procedures performed until distanceinformation is output in the image processing apparatus shown in FIG.37;

FIG. 39 is a flow chart showing procedures including a process ofselecting a window on the basis of a type of a road on which a vehicleruns in a window switching process;

FIG. 40 is a diagram showing an example of a freeway window selected bythe window switching process;

FIG. 41 is a diagram showing an example of a standard window selected bythe window switching process in FIG. 39;

FIG. 42 is a flow chart showing procedures including a process ofselecting a window on the basis of tendency of a gradient of a road onwhich a vehicle runs in the window switching process shown in FIG. 38;

FIG. 43 is a diagram showing an example of a concave-surface windowselected by the window switching process in FIG. 42;

FIG. 44 is a diagram showing an example of a convex-surface windowselected by the window switching process in FIG. 42;

FIG. 45 is a block diagram showing a configuration of an imageprocessing apparatus according to an eighth embodiment of the presentinvention;

FIG. 46 is a block diagram showing a configuration of an imageprocessing apparatus according to a ninth embodiment of the presentinvention;

FIG. 47 is a block diagram showing a configuration of an imageprocessing apparatus according to a tenth embodiment of the presentinvention;

FIG. 48 is a flow chart showing an outline of an image processing methodaccording to the tenth embodiment of the present invention;

FIG. 49 is a diagram showing an output of a distance image;

FIG. 50 is a diagram showing an image processing method depending on acombination between a distance zone and a speed zone;

FIG. 51 is a diagram showing a display obtained when image processingrelated to vehicle detection;

FIG. 52 is a diagram showing a display obtained when image processingrelated to white line detection;

FIG. 53 is a diagram showing another example of an image processingmethod depending on a combination between a distance zone and a speedzone;

FIG. 54 is a block diagram showing a configuration of an imageprocessing apparatus according to one modification of the tenthembodiment of the present invention;

FIG. 55 is a block diagram showing a partial configuration of an imageprocessing apparatus according to an eleventh embodiment of the presentinvention; and

FIG. 56 is a diagram showing an example of an image picked by an imagingunit shown in FIG. 55.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Best modes (to be referred to as embodiments hereinafter) for carryingout the present invention will be described below with reference to theaccompanying drawings.

FIG. 1 is a block diagram showing a configuration of an image processingapparatus according to a first embodiment of the present invention. Animage processing apparatus 1 shown in FIG. 1 includes: an imaging unit10 which has a predetermined field of image pickup field and picks animage corresponding to the field of image pickup field to generate animage signal group; an image analyzing unit 20 which analyzes an imagesignal group generated by the imaging unit 10 to process the imagesignal group; a control unit 30 which controls all processes andoperations of the image processing apparatus 1; an output unit 40 whichoutputs various pieces of information including distance information; astorage unit 50 which stores the various pieces of information includingdistance information; and a detection unit 60 which detects movinginformation of a given vehicle serving as a mobile object such as agiven vehicle on which the image processing apparatus 1 is mounted. Theimaging unit 10, the image analyzing unit 20, the output unit 40, thestorage unit 50, and the detection unit 60 are electrically connected tothe control unit 30. Modes of the connection include not only a wiredmode but also a wireless mode.

The imaging unit 10 is a stereo camera having a right camera 11 a and aleft camera 11 b which are arranged on the right and left side by side.The right camera 11 a includes a lens 12 a, an image pickup element 13a, an analog/digital (A/D) conversion unit 14 a, and a frame memory 15a. The lens 12 a converges light from an arbitrary object arranged in apredetermined field of image pickup view on the image pickup element 13a. The image pickup element 13 a is an image pickup element such as aCCD or a CMOS, detects light from the object converted by the lens 12 aas an optical signal, converts the optical signal into an electricsignal serving as an analog signal to output the electric signal. TheA/D conversion unit 14 a converts the analog signal output from theimage pickup element 13 a into a digital signal to output the digitalsignal. The frame memory 15 a stores the digital signal output from theA/D conversion unit 14 a and arbitrarily outputs a digital signal groupcorresponding to one image pickup image as an image signal groupcorresponding to the field of image pickup view, i.e., imageinformation. On the other hand, the left camera 11 b has the sameconfiguration as that of the right camera 11 a and includes a lens 12 b,an image pickup element 13 b, an A/D conversion unit 14 b, and a framememory 15 b. The respective constituent components of the left camera 11b have the same functions as the corresponding constituent components ofthe right camera 11 a.

The lenses 12 a and 12 b serving as one pair of image pickup opticalsystems held by the imaging unit 10 are separately located with adistance L to have parallel optical axes. The image pickup elements 13 aand 13 b are separately located with distances f from the lenses 12 aand 12 b on the optical axes, respectively. The right camera 11 a andthe left camera 11 b picks images of the same object from differentpositions through different optical paths, respectively. The lenses 12 aand 12 b are generally constructed by combining a plurality of lenses.For example, the lenses 12 a and 12 b preferably correct aberrationssuch as distortions of the lenses.

The image analyzing unit 20 includes a distance calculation unit 21which processes an image signal group acquired from the imaging unit 10to calculate a distance to the object the image of which is picked. Thedistance calculation unit 21 detects a right image signal matched withan arbitrary left image signal in a left image signal group output bythe left camera 11 b from a right image signal group output by the rightcamera 11 a and calculates a distance to an object located in the fieldof image pickup view on the basis of an amount of movement which is adistance between the detected right image signal and a correspondingleft image signal. More specifically, positions of a right image signalgroup from the right camera 11 a and a left image signal group from theleft camera 11 b on optical axes of image pickup optical systems areoverlapped on a reference, and an arbitrary left image signal in theleft image signal group and a right image signal in the right imagesignal group maximally matched with the left image signal are detected,and an amount of movement I which is a distance from the correspondingleft image signal to the right image signal on the image pickup elementis obtained. By using the following equation (1) based on the principleof triangular surveying, for example, a distance R from the imaging unit10 to a vehicle C in FIG. 1 is calculated. In fact, the amount ofmovement I is preferably obtained on the basis of the number of pixelsand a pixel pitch of an image pickup element.

For the descriptive convenience, the parallel stereo is described above.However, crossing of the axes at an angle, different focal distances,different positional relationships between the image pickup elements andthe lenses, and the like are calibrated and corrected by rectification,so that a parallel stereo may be realized by calculation processing.R=f·L/I  (1)

The distance calculation unit 21 calculates a distance to an objectcorresponding to an arbitrary image signal in an calculation range,i.e., an object corresponding to an arbitrary pixel of the image pickupelement. The image analyzing unit 20 forms distance information in whicha distance to be calculated to the object and a position of an object inan image are caused to correspond to each other by the distancecalculation unit 21 and output the distance information to the controlunit 30. The object the distance of which is calculated here is notlimited to an object serving as a tangible entity but also an arbitraryobject the image of which is to be picked and which includes a roadsurface or a background such as sky.

The control unit 30 includes a CPU which executes a processing programstored in the storage unit 50 to control various processing operationsin the imaging unit 10, the image analyzing unit 20, the output unit 40,the storage unit 50, and the detection unit 60. In particular, thecontrol unit 30 according to the first embodiment includes an arrivalpoint prediction unit 31 and an calculation range setting unit 32. Thearrival point prediction unit 31 acquires moving information from thedetection unit 60 to predict a point which a given vehicle reaches apredetermined period of time after. the calculation range setting unit32 selects a window matched with a prediction result of the arrivalpoint prediction unit 31 from pieces of window information stored inwindow information 51 to set an calculation range in the distancecalculation unit 21. The calculation range setting unit 32 outputs theselected window information to the image analyzing unit 20. The windowinformation is information related to a size, a shape, and the like of awindow.

The output unit 40 outputs various pieces of information includingdistance information. For example, the output unit 40 includes a displayapparatus such as a liquid crystal display or an organic EL(Electroluminescence) display to display various displayable pieces ofinformation such as an image or the like picked by the imaging unit 10together with the distance information. Furthermore, the output unit 40includes an audio output apparatus such as a loudspeaker, and may beconstructed to output various pieces of audio information such as awarning based on the distance information or the distance information.

The storage unit 50 includes a ROM in which various pieces ofinformation such as a program and an image processing program forstarting a predetermined OS is stored in advance and a RAM in whichcalculation parameters for processing and various pieces of informationinput/output to various constituent components are stored. Furthermore,the storage unit 50 stores a window information 51 which stores piecesof window information selected by the calculation range setting unit 32,image information 52 obtained by an image pickup operation by theimaging unit 10, steering angle information 53 detected by a steeringangle sensor 61, speed information 54 detected by a speed sensor 62,arrival point information 55 predicted by the arrival point predictionunit 31, and distance information 56 calculated and formed by thedistance calculation unit 21.

The image processing program described above can be recorded on acomputer readable recording medium such as a hard disk, a flexible disk,a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, DVD±R, a DVD±RW, a DVD-RAM, a MOdisk, a PC card, an xD picture card, or a smart media and can be widelycirculated.

The detection unit 60 detects moving information of a given vehicle. Inparticular, the detection unit 60 in the first embodiment includes thesteering angle sensor 61 which detects a moving direction of the givenvehicle and a speed sensor 62 which detects a moving speed. The steeringangle sensor 61 is a sensor which detects a steering angle serving asright and left rotating angles of front wheels as a moving direction ofthe given vehicle and which calculates a steering angle on the basis ofa rotating angle and a rotating direction of a steering wheel. The speedsensor 62 is a speedometer that such as an automobile generallyprovides. The detection unit 60 outputs the steering angle informationdetected by the steering angle sensor 61 and the speed informationdetected by the speed sensor 62 to the control unit 30 as the movinginformation of the given vehicle. The speed sensor 62 may be a sensorwhich observes a road surface on which the given vehicle runs tocalculate a speed, a sensor which detects an acceleration to calculate aspeed, or the like. As a means which detects a moving direction of thegiven vehicle, in place of the steering angle sensor 61, an angledetection sensor using a gyroscope may be used.

In the first embodiment, the detection unit 60 constitutes at least apart of a moving information detection unit 600 which detects movinginformation including a speed of a mobile object. The calculation rangesetting unit 32 constitutes at least a part of a processing contentsetting unit 320 which sets contents of processes to be performed in animage based on the moving information detected by the moving informationdetection unit 600. Furthermore, the distance calculation unit 21constitutes at least a part of a processing calculation unit 210 whichperforms processing calculation corresponding to the contents of theprocesses set by the processing content setting unit 320.

A processing operation executed by the image processing apparatus 1 willbe described below with reference to a flow chart in FIG. 2. FIG. 2 is aflow chart showing procedures performed until the image processingapparatus 1 outputs distance information after an image is picked.

As shown in FIG. 2, the imaging unit 10 performs an image pickup processwhich picks a predetermined field of view and outputs a generated imagesignal group to the image analyzing unit 20 as image information (stepS101). More specifically, in the right camera 11 a and the left camera11 b in the imaging unit 10, under the control of the control unit 30,lights are converted from a region included in a predetermined fieldangle by using the lenses 12 a and 12 b.

The lights converged by the lenses 12 a and 12 b are focused on thesurfaces of the image pickup elements 13 a and 13 b and converted intoelectric signals (analog signals), respectively. The analog signalsoutput from the image pickup elements 13 a and 13 b are converted intodigital signals by the A/D conversion units 14 a and 14 b, respectively,and the converted digital signals are temporarily stored in the framememories 15 a and 15 b, respectively. The digital signals temporarilystored in the frame memories 15 a and 15 b are sent to the imageanalyzing unit 20 a predetermined period of time after.

FIG. 3 is an explanatory diagram conceptually showing an image pickupprocess by a fly-eye stereo camera. FIG. 3 shows a case in which anoptical axis z_(a) of the right camera 11 a and an optical axis z_(b) ofthe left camera 11 b are parallel to each other. In this case, a pointcorresponding to a point A_(b) in a left image region I_(b)corresponding to a coordinate system (left camera coordinate system)inherent in the left camera is present on a straight line α_(E)(epipolar line) in a right image region I_(a) corresponding to acoordinate system (right camera coordinate system) inherent in the rightcamera. FIG. 3 shows a case in which a corresponding point in the rightcamera 11 a is searched for with reference to the left camera 11 b. Incontrast to this, the right camera may be used as a reference.

After the image pickup process in step S101, the detection unit 60performs a moving information detection process which detects movinginformation of the given vehicle to output the moving information to thecontrol unit 30 (step S103). In this case, in the detection of themoving information, the steering angle sensor 61 detects a steeringangle as a moving direction, and the speed sensor 62 detects a movingspeed. The arrival point prediction unit 31 of the control unit 30performs an arrival point prediction process which an arrival point ofthe given vehicle a predetermined period of after on the basis of themoving information from the detection unit 60 (step S105). Thecalculation range setting unit 32 performs an calculation range settingprocess which sets an calculation range to be processed by the distancecalculation unit 21 on the basis of a prediction result of the arrival(step S107). The control unit 30 outputs information of the setcalculation range to the image analyzing unit 20.

In the image analyzing unit 20, the distance calculation unit 21calculates a distance to an arbitrary object the image of which ispicked within the set calculation range, forms distance information inwhich the calculated distance and the position of the object in theimage are caused to correspond to each other, and outputs the distanceinformation to the control unit 30 (step S109).

In order to perform the distance calculation in step S109, coordinatevalues of all pixel points or some pixel points in the field of view theimage of which is picked by using the right and left camera coordinatesystems. However, prior to this calculation, calculation of coordinatevalues in the left and right camera coordinate system and corresponding(corresponding point searching) between the coordinate values areperformed. When a three-dimension is reconstructed by the correspondingpoint searching, a pixel point located on an arbitrary straight linepassing through an image serving as a reference is desirably located onthe same straight line in the other image (epipolar condition ofconstraint). However, the epipolar condition of constraint is not alwayssatisfied. For example, in a stereo image region I_(ab) shown in FIG. 4,a point of the right image region I_(a) corresponding to the point A_(b)in the left image region I_(b) serving as a reference is present on astraight line α_(A), and a point in the right image region I_(a)corresponding to a point B_(b) in the left image region I_(b).

When the epipolar condition of constraint is not satisfied, a searchingrange cannot be narrowed down, and an amount of calculation in searchingof a corresponding point is vast. In such a case, the image analyzingunit 20 performs a process (rectification process) which normalizes theright and left camera coordinate systems in advance to convert the stateinto a state in which the epipolar condition of constraint is satisfied.FIG. 5 shows a corresponding relationship between the right and leftimage regions after the rectification process. As shown in FIG. 5, whenthe epipolar condition of constraint is satisfied, searching can beperformed such that the searching range is narrowed to the epipolar line(X. For this reason, an amount of calculation required for searching acorresponding point can be reduced.

An example of a corresponding point searching method will be describedbelow. In the referenced left image region I_(b), a local region is setnear an interested pixel point, and the same region as the local regionis set on the corresponding epipolar line (E in the right image regionI_(a). A local region having the highest degree of similarity to thelocal region in the left image region I_(b) is searched for while thelocal region in the right image region I_(a) is scanned on the epipolarline α_(E). As a result of the searching, a center point of a localregion having the highest degree of similarity is set as a correspondingpoint of a pixel point in the left image region I_(b).

As the degree of similarity used in the corresponding point searching, asum (SAD: Sum of Absolute Difference) of absolute values of differencesbetween pixel points in the local region, a sum of squares (SSD: Sum ofSquared Difference) of differences between pixel points in the localregion, or normalized cross correlation (NCC) between pixel points inthe local region can be applied. When the SAD or the SSD of thesedegrees of similarity is applied, a point having a minimum value is setas a point having the highest degree of similarity. When the NCC isapplied, a point having a maximum value is set as a point having thehighest degree of similarity.

Subsequent to step S109 described above, the control unit 30 outputs thedistance information and predetermined processing information based onthe distance information to the output unit 40 (step S111), and oneseries of processing operations are finished. The control unit 30arbitrarily stores the image information 52, the steering angleinformation 53 the speed information 54, the arrival point information55, and the distance information 56 serving as pieces of informationgenerated in the various processing steps in the storage unit 50.

The series of processes are continuously repeated in the absence of adesignation of a predetermined end of processes or a designation ofinterruption from a passenger or the like of a given vehicle. Theprocesses described above are described as processes to be sequentiallyexecuted. However, in fact, independent processing operations areparallel performed by a processor to increase a speed of a processingcycle. The image pickup process in step S101 may be performedimmediately after any one of the processes in step S103, step S105, andstep S107.

An arrival point prediction process in step S105 shown in FIG. 2 will bedescribed below. FIG. 6 is a flow chart showing procedures of thearrival point prediction process performed by the arrival pointprediction unit 31. As shown in FIG. 6, the arrival point predictionunit 31 acquires moving information from the detection unit 60 (stepS122) and calculates an arrival point of the given vehicle after apredetermined period of time (step S124). In this calculation, thearrival point prediction unit 31 calculates an arrival point in acoordinate system of a real space. Thereafter, the position is convertedinto coordinates to the coordinate system on an image pickup image (stepS126), the converted coordinates of the arrival point are output to thecalculation range setting unit 32 as arrival point information (stepS128), and the arrival point prediction process is ended.

In the calculation of the arrival point in step S124, the arrival pointprediction unit 31 calculates coordinates of an arrival point after apredetermined period of time on the assumption that the given vehiclelinearly moves at a speed detected by the speed sensor 62 in a movingdirection detected by the steering angle sensor 61. For example, asshown in FIG. 7, when a given vehicle Ca runs on a driving road R curvedto the right, the arrival point prediction unit 31 calculatescoordinates of an arrival point p1 after a predetermined period of timeon the assumption that the given vehicle goes linearly at a speeddetected by the speed sensor 62 in a moving direction detected by thesteering angle sensor 61 at a present point p0, i.e., a direction havinga steering angle of θ0. Thereafter, the arrival point prediction unit 31converts an arrival point p1 in the coordinate system in the real spaceinto an arrival point P1 corresponding to the coordinate system on animage 16 as shown in FIG. 8, and outputs the coordinates as arrivalpoint information.

Subsequently, an calculation range setting process in step S107 shown inFIG. 2 will be described below. FIG. 9 is a flow chart showingprocedures of the calculation range setting process performed by thecalculation range setting unit 32. As shown in FIG. 9, the calculationrange setting unit 32 acquires arrival point information from thearrival point prediction unit 31 (step S142). Thereafter, thecalculation range setting unit 32 performs a process of setting ancalculation range on the basis of the acquired arrival point information(steps S144 to S150). More specifically, the calculation range settingunit 32 determines whether the arrival point is close to the givenvehicle (step S144). When the arrival point is close to the givenvehicle (step S144: Yes), a short-distance window which is maximallymatched with the arrival point information is selected as an imageprocessing window which regulates an calculation range from the windowinformation 51 stored in the storage unit 50 (step S146), and the windowinformation of the selected short-distance window is output to the imageanalyzing unit 20 as information of the calculation range (step S148).On the other hand, when the arrival point is far from the given vehicle(step S144: No), the calculation range setting unit 32 selects along-distance window which is maximally matched with the arrival pointinformation from the window information 51 (step S150) and outputs thewindow information to the image analyzing unit 20 (step S148).

An example of a window selected by the calculation range setting unit 32will be described below. FIG. 10 is a diagram obtained when an arrivalpoint P1 a predicted by the arrival point prediction unit 31 and ashort-distance window 51 a selected by the calculation range settingunit 32 are overlapped on the image 16 the image of which is picked bythe imaging unit 10. The short-distance window 51 a is, as shown in FIG.10, an approximately trapezoidal polygonal window including the arrivalpoint P1 a therein. The short-distance window 51 a is a window selectedwhen the arrival point P1 a is relatively close to the given vehicle andincludes a periphery of the arrival point P1 a and a periphery of aroute extending from the given vehicle to the arrival point P1 a. In theshort-distance window 51 a, a height of the approximately trapezoidalshape and a length of the upper base are regulated by a distance to thearrival point P1 a, and a lateral position of the upper base isregulated by a direction to the arrival point P1 a.

On the other hand, FIG. 11 is a diagram obtained by overlapping anarrival point P1 b on a long-distance window 51 b. The long-distancewindow 51 b, as shown in FIG. 11, is a rectangular window including aperipheral region around the arrival point P1 b. The window is selectedwhen the arrival point P1 b is located at a position relatively far fromthe given vehicle. On the long-distance window 51 b, a distance to thearrival point P1 b regulates a window size.

The window information 51 held by the storage unit 50 stores pieces ofwindow information regulated by using a direction and a distance to apredicted arrival point as parameters. The calculation range settingunit 32 selects the short-distance window 51 a, the long-distance window51 b, or the like as a window which is maximally matched with thepredicted arrival point from the pieces of window information. At thistime, the calculation range setting unit 32, for example, may store acorresponding table between parameters and window information and referto the corresponding table in accordance with various combinations ofthe parameters to select a window to be selected.

In the flow chart shown in FIG. 9, a short-distance window or along-distance window is selected. However, a choice may be added toselect, for example, an intermediate-distance window. One example ofthis is shown in FIG. 12. An intermediate-distance window 51 c shown inFIG. 12 includes a periphery of an arrival point P1 c located betweenthe short-distance point and the long-distance point from the givenvehicle and a periphery of a portion on a route extending from the givenvehicle to the arrival point P1 c except for a range close to the givenvehicle. The number of choices is not limited to the number in theexample, and windows of various types corresponding to the large numberof choices are set. Different windows may be substantially designed tobe selected for an arbitrary combination of parameters.

The image processing apparatus 1 according to the first embodimentdescribed above predicts an arrival point of a given vehicle after apredetermined period of time on the basis of steering angle informationand speed information serving as moving information and sets an limitsan calculation range depending on the predicted arrival point tocalculate a distance. For this reason, in comparison with a conventionalimage processing apparatus which calculates a distance to all imagesignals of image information, a time required for distance calculationcan be shortened. As a result, the image processing apparatus 1 canshorten processing time required until distance information is outputafter image information is acquired, and the distance information can beoutput at a high speed.

A second embodiment of the present invention will be described below. Inthe first embodiment described above, in the arrival point predictionprocess in step S105, the arrival point prediction unit 31 calculates anarrival point of a given vehicle a predetermined period of time aftersuch that the given vehicle moves linearly at a speed detected by thespeed sensor 62 in a moving direction detected by the steering anglesensor 61. However, in the second embodiment, with reference to a changerate of a steering angle serving as a change rate of a moving direction,an arrival point is calculated by using a function which isexperientially calculated.

FIG. 13 is a block diagram showing a configuration of an imageprocessing apparatus according to the second embodiment of the presentinvention. An image processing apparatus 2 shown in FIG. 13 includes acontrol unit 130 in place of the control unit 30 of the image processingapparatus 1 according to the first embodiment. The control unit 130includes, in place of the arrival point prediction unit 31 and thecalculation range setting unit 32 included in the control unit 30, anarrival point prediction unit 33 and an calculation range setting unit34 (at least a part of a processing content setting unit 340) having thesame functions as those of the arrival point prediction unit 31 and thecalculation range setting unit 32. The image processing apparatus 2includes a storage unit 150 in place of the storage unit 50. the storageunit 150 further stores prediction function information 57 in which thestorage contents in the storage unit 50 and a plurality of functionsselected by the arrival point prediction unit 33. The remainingconfiguration is the same as that in the first embodiment. The samereference numerals as in the first embodiment denote the same parts inthe second embodiment.

The image processing apparatus 2 executes an image pickup process to aprocess of outputting distance information shown in the flow chart inFIG. 2. With respect to a prediction method of an arrival point in thearrival point prediction process and characteristics of a windowselected in an calculation range setting process, the image processingapparatus 2 is different from the image processing apparatus 1.

An arrival point prediction process executed by the image processingapparatus 2 will be described below. FIG. 14 is a flow chart showingprocedures of an arrival point prediction process performed by thearrival point prediction unit 33. As shown in FIG. 14, the arrival pointprediction unit 33 acquires steering angle information and speedinformation as moving information from the detection unit 60 (stepS222), calculates a change rate of a steering angle (step S224), selectsan appropriate prediction function from the prediction functioninformation 57 stored in the storage unit 150 on the basis of thesteering angle, the change rate of the steering angle, and the speed(step S226), and calculates an arrival point of a given vehicle apredetermined period of time after according to the selected predictionfunction (step S228). Furthermore, the arrival point prediction unit 33performs conversion of coordinates as in step S126 in the firstembodiment with respect to the calculated arrival point (step S230), andoutputs the converted coordinates of the arrival point to thecalculation range setting unit 34 as arrival point information (stepS132) to end the arrival point prediction process.

The prediction function selected by the arrival point prediction unit 33in step S226 is a function experientially calculated and is a functionregulated by using the steering angle, the change rate of the steeringangle, and the speed. Furthermore, functionally, the prediction functionis a function which predicts a track of the given vehicle, a track drawnby the prediction function by using time as a variable expresses aprediction track of the given vehicle. For example, as shown in FIG. 15,a prediction function selected at a present point p0 of a given vehicleCa draws a track indicated by a broken line by using time as a variable.The arrival point prediction unit 33 recognizes prediction track q ofthe given vehicle Ca after the present point p0. In step S228,coordinates of an arrival point p2 a predetermined period of time afteron the track.

In this case, a relationship between the prediction function selected bythe arrival point prediction unit 33 and the steering angle, the changerate of the steering angle, and the speed serving as parameters forregulating the prediction function will be described below withreference to FIG. 16. FIG. 16 shows a state in which the given vehicleCa moves from the present point p0 at a speed v0 in a direction having asteering angle θ0. The change rate of the steering angle calculated instep S224 is a difference Δθ0 obtained by subtracting a steering angledetected immediately before the present point p0 from the steering angledetected at the present point p0. For example, when an absolute value ofthe difference Δθ0 is 0, the arrival point prediction unit 33 predictsthat the given vehicle Ca continuously moves while keeping the presentsteering angle and selects a prediction function corresponding to aprediction track q0 obtained at this time. On the other hand, when thedifference Δθ0 is a positive value, the arrival point prediction unit 33predicts that the given vehicle Ca moves while increasing the steeringangle, and selects a prediction function corresponding to a predictiontrack q1 having a curvature radius smaller than that of the predictiontrack q0. On the other hand, when the difference Δθ0 is a negativevalue, the arrival point prediction unit 33 predicts that the givenvehicle Ca moves while decreasing the steering angle, and selects aprediction function corresponding to a prediction track q2 having acurvature radius larger than that of the prediction track q0. The sizesof the curvature radius of the prediction tracks q1 and q2 arecalculated with reference to the size of the difference Δθ0 by using thecurvature radius of the prediction track q0 as a standard.

As another case, for example, when a steering angle at the present pointp0, a change rate of the steering angle, and the speed are given by θ0,θθ0, and v0, respectively, the prediction track q0 is applied. When thespeed is given by v1 (>v0), the prediction track q1 having a curvatureradius smaller than that of the prediction track q0 is applied. When thespeed is v2 (<v0), the prediction track q2 having a curvature radiuslarger than that of the prediction track q0 is applied. Furthermore,depending on a combination of the steering angle, the change rate of thesteering angle, and the speed, various prediction functions can beregulated in various cases.

A window selected by the calculation range setting unit 34 in thecalculation range setting process will be described below. FIG. 17 is adiagram obtained by overlapping an arrival point P2 a calculated by thearrival point prediction unit 33, a selected prediction route Qa, and ashort-distance window 51 d selected by the calculation range settingunit 34 from the window information 51 on the image 16 picked by theimaging unit 10. The short-distance window 51 d corresponds to theshort-distance window 51 a described in the first embodiment and issimilar to the short-distance window 51 a since the short-distancewindow 51 d includes a periphery of the arrival point P2 a and aperiphery of the prediction route Qa extending from the given vehicle tothe arrival point P2 a. The short-distance window 51 d is different fromthe short-distance window 51 a in that a boundary between the right andleft windows is formed by a bent line along the prediction route Qa. Asthe bent line, a curved line may be used.

On the other hand, FIG. 18 is a diagram obtained by an arrival point P2b, a prediction route Qb, and a long-distance window 51 e on the image16. The long-distance window 51 e is a rectangular window including aregion around the arrival point P2 b, and is similar to thelong-distance window 51 b described in the first embodiment. Thecalculation range setting unit 34 may store a corresponding tableconstituted by, for example, an arrival point, a predicted route or aprediction function, and window information and refer to thecorresponding table depending on various combinations of these elementsto determine a window to be selected.

In the prediction for the arrival point in the second embodiment, anarrival point a predetermined period of time after is obtained by usinga prediction function experientially determined to predict a track andusing a steering angle, a change rate of the steering angle, and a speedat a present point as parameters. For this reason, in comparison withprediction for an arrival point based on linear movement in the firstembodiment, prediction can be performed with higher reliability. Adifference between arrival points predicted by both the methods is shownin FIG. 17 such that the predicted arrival points are shown in the sameimage. The arrival point P2 a predicted on the basis of the curvedprediction track Qa is predicted as a position which is out of thelinearly predicted arrival point P1 a to the lower left side. As shownin the same image in FIG. 18, in the long-distance region, the arrivalpoint P2 b predicted on the basis of the prediction route Qb ispredicted as a position which is out of the linearly predicted arrivalpoint P1 b to the lower right side in the image.

In the second embodiment, a track is predicted by using a predictionfunction based on a rule of thumb. In particular, in calculation in theshort-distance region, since a boundary between the right and leftwindows is formed by a bent line or a curved line on the basis of theprediction track, an calculation range can be more strictly limited, andan unnecessary calculation region can be omitted. As a result,processing time taken for calculation can be further shortened.

In embodiment 2, the prediction function is selected by using thesteering angle, the change rate of the steering angle, and the speed asparameters. However, parameters related to the selection are not limitedto the parameters mentioned above. For example, an acceleration may beadded as a parameter, or an acceleration may be used in place of thechange rate of the steering angle to select a prediction function.Furthermore, an orientation of movement may be additionally detected inplace of the steering angle and used as a parameter.

In the image processing apparatus 2 according to the second embodimentdescribed above, an arrival point and prediction track are strictlypredicted by using the prediction function, and, accordingly, thecalculation range is set and limited to calculate a distance. For thisreason, time required for distance calculation can be shortened incomparison with a conventional image processing apparatus which performsdistance calculation to all image signals of image information. As aresult, the image processing apparatus 2 can shorten processing timerequired until distance information is output after the imageinformation is acquired and output the distance information at a highspeed.

A third embodiment of the present invention will be described above. Inthe first and the second embodiments described above, an arrival pointprediction unit calculates an arrival point of a given vehicle apredetermined period of time after on the basis of steering angleinformation and speed information. However, in the third embodiment,position information of the given vehicle is detected, and an arrivalpoint a predetermined period of time after is calculated with referenceto map information serving as geographical information near a presentposition.

FIG. 19 is a block diagram showing a configuration of an imageprocessing apparatus according to the third embodiment of the presentinvention. The image processing apparatus 3 shown in FIG. 19 includes adetection unit 160 (at least a part of a moving information detectionunit 1600) obtained by adding a GPS (Global Positioning System) sensor63 to the detection unit 60 of the image processing apparatus 1according to the first embodiment. The image processing apparatus 3includes a control unit 230 in place of the control unit 30. The controlunit 230 includes, in place of the arrival point prediction unit 31 andthe calculation range setting unit 32, an arrival point prediction unit35 and an calculation range setting unit 36 (part of a processingcontent setting unit 360) having the same functions as that of thearrival point prediction unit 31 and the calculation range setting unit32. Furthermore, the image processing apparatus 3 includes a storageunit 250 in place of the storage unit 50. The storage unit 250 stores,in addition to storage contents of the storage unit 50, positioninformation 58 serving as position information detected by the GPSsensor 63 and map information 59 referred to on the basis of thedetected position information. The remaining configuration is the sameas that in the first embodiment. The same reference numerals as in thefirst embodiment denote the same parts in the second embodiment.

As in the procedures of the image processing apparatus 1 described inthe first embodiment, the image processing apparatus 3 executes an imagepickup process to a process of outputting distance information shown inthe flow chart in FIG. 2. The image processing apparatus 3 is differentfrom the image processing apparatus 1 in that position information isfurther detected in the moving information detection process and that anarrival point is corrected on the basis of map information in thearrival point prediction process.

The moving information detection process executed by the imageprocessing apparatus 3 will be described below. In the detection unit160, as in the detection unit 60 of the image processing apparatus 1,the steering angle sensor 61 detects a steering angle, and the speedsensor 62 detects a speed. Thereafter, the GPS sensor 63 detects apresent position of a given vehicle. Each of the sensors outputs adetected result to the control unit 230, respectively.

An arrival point prediction process executed by the image processingapparatus 3 will be described below. The arrival point prediction unit35, like the arrival point prediction unit 31 of the image processingapparatus 1, calculates coordinates of an arrival point after apredetermined period of time on the basis of the steering angleinformation and the speed information on the assumption that the givenvehicle linearly moves. Thereafter, the arrival point prediction unit 35refers to the map information 59 stored by the storage unit 250 on thebasis of the position information detected by the GPS sensor 63 tocorrect coordinates of the calculated arrival point. The map informationto be referred to is peripheral geographical information including apresent position, for example, the map information includes informationof a width, inclination of a road on which the vehicle runs, a curvatureof a curve, and the like.

As show in FIG. 20, when it is detected by the referred map informationthat the given vehicle approaches to an ascending slope, the arrivalpoint prediction unit 35 converts an arrival point p3 a calculated by alinear track into an arrival point p3 b having a distance equal to thearrival point p3 a along the ascending slope. In this conversion, thearrival point prediction unit 35 corrects the coordinates of the arrivalpoint by Δpz in the moving direction and Δpy in the vertical direction.In this case, a position of a window set in the calculation rangesetting process is vertically corrected on the pickup image.

In the above description, a track is corrected on the basis ofinformation of the inclination of the ascending slope to correct theposition of the arrival point. However, correction may be performed inconsideration of a decrease in speed caused by running on the ascendingslope. When it is detected that the given vehicle runs on a descendingslope, correction may be performed in consideration of an increase inspeed. Furthermore, various corrections can be performed on the basis ofvarious pieces of map information. When the various corrections areperformed, in the image processing apparatus 3, an arrival point can bepredicted with higher reliability without being influenced by ageographical condition of a driving road. As described in the secondembodiment, by using a change rate of a steering angle serving as achange rate of a moving direction of the given vehicle, an arrival pointmay be predicted on the basis of a rounded track.

In the image processing apparatus 3 according to the third embodimentdescribed above, position information is acquired by using a GPS, and ancalculation range is set and limited depending on the arrival pointcorrected on the basis of the map information corresponding to theacquired position information. For this reason, in comparison with aconventional image processing apparatus which performs distancecalculation to all image signals of image information, time required forthe distance calculation can be shortened. As a result, the imageprocessing apparatus 3 can shorten processing time required until thedistance information is output after the image information is acquired,and can output the distance information at a high speed.

In the third embodiment, on the basis of the image processing apparatus1, by further using the GPS, an arrival point is corrected withreference to the map information corresponding to the positioninformation. However, an calculation method using the predictionfunction described in the image processing apparatus 2 according to thesecond embodiment described above may be further applied to perform anarrival point prediction process. Furthermore, map information may beacquired from the outside of the apparatus by using a GPS to correct aprediction track on the basis of the acquired map information.

A fourth embodiment of the present invention will be described below. Inthe first to the third embodiments described above, a distance to apicked object is detected by processing an image signal group from theimaging unit 10. However, in the fourth embodiment, a distance to anobject located in a field of image pickup view is detected by a radar.

FIG. 21 is a block diagram showing a configuration of an imageprocessing apparatus according to the fourth embodiment of the presentinvention. An image processing apparatus 4 shown in FIG. 21 includes theimage processing apparatus 1 according to the first embodiment describedabove, a radar 70, and a control unit 330 having a function ofcontrolling the radar 70 in place of the control unit 30. The remainingconfiguration is the same as that in the first embodiment. The samereference numerals as in the first embodiment denote the same parts inthe fourth embodiment. In the fourth embodiment, the detection unit 60and the radar 70 constitute at least a part of a moving informationdetection unit 670.

The radar 70 transmits a predetermined outgoing wave, receives areflected wave obtained by reflecting the outgoing wave by an objectsurface, and detects a distance from the radar 70 to the object whichreflects the outgoing wave and a direction in which the object islocated on the basis of an outgoing state and a receiving state. Theradar 70 detects a distance to the object which reflects the outgoingwave and a direction of the reflection on the basis of an outgoing angleof the outgoing wave, an incident angle of the reflected wave, areceiving intensity of the reflected wave, time until the reflected waveis received after the outgoing wave is transmitted, and changes infrequency in the received wave and the reflected wave. The radar 70outputs the distance to the object located in the field of image pickupview of the imaging unit 10 to the control unit 330 together with thedirection in which the object is located. The radar 70 transmits, forexample, a laser beam, an infrared ray, a millimeter wave, a microwave,an ultrasonic wave, or the like as a transmitter wave.

In the image processing apparatus 4 according to the fourth embodiment,since a distance is detected by the radar 70 in place of calculation ofa distance by processing image information from the imaging unit 10, thedistance information can be acquired at a higher speed and a highaccuracy.

In the image processing apparatus 4, matching between a positionalrelationship in image signal group picked by the imaging unit 10 and apositional relationship in a detection range of the radar 70 is obtainedin advance as described below to perform processes. For example, theimage processing apparatus 4 performs an image pickup process by theimaging unit 10 and a detection process by the radar 70 to an objecthaving a known shape, positions of the known object processed by theimaging unit 10 and the radar 70 are obtained. Thereafter, the imageprocessing apparatus 4 uses the method of least squares or the like toobtain a relationship between the positions of the known objectsprocessed by the imaging unit 10 and the radar 70, and matches thepositional relationship in the image signal group picked by the imagingunit 10 and the positional relationship in the detection range of theradar 70.

In the image processing apparatus 4, even though an original point ofdetection of the imaging unit 10 and an original point of detection ofthe radar 70 are different from each other, if distances from the imagepickup point and the detection point to the image processing apparatus 4are sufficiently long, it can be understood that an image pickuporiginal point and a detection original point are almost overlapped.Furthermore, when matching between the positional relationship in theimage signal group picked by the imaging unit 10 and the positionalrelationship in the detection range of the radar 70 is accuratelyperformed, the difference between the image pickup original point andthe detection original point can be corrected by geometric conversion.

In the image processing apparatus 4, radar detection points of the radar70 are preferably set to be located at predetermined intervals in apixel row in which image signals of the image signal group picked by theimaging unit 10. When the radar detection points are not set asdescribed above, interpolation points of the radar detection points areobtained in the same pixel row as the row of the image signals by usingprimary interpolation or the like on the basis of a plurality of radardetection points located near the image signals, and a detection processmay be performed by the interpolation points.

In the image processing apparatuses according to the first to the fourthembodiments, an arrival point a predetermined period of time after ispredicted by using moving information detected at a present point.However, a future arrival point, a future prediction track, or the likemay be predicted on the basis of arrival point information, movinginformation, and the like of time-series stored in the storage unit, orcorrection may be performed.

FIG. 22 is a block diagram showing a configuration of an imageprocessing apparatus according to a fifth embodiment of the presentinvention. An image processing apparatus 5 includes an imaging unit 10which picks a predetermined field of view, an image analysis unit 520which analyzes an image generated by the imaging unit 10, a control unit530 which performs operation control of the image processing apparatus5, an output unit 40 which outputs information such as an image orletters to display the information, a storage unit 550 which storesvarious data, and a probing unit 560 which probes moving information ofa given vehicle serving as a mobile object such as a four-wheel vehicleon which the image processing apparatus 5 is mounted. The same referencenumerals as in the image processing apparatus 1 according to the firstembodiment denote the same parts in the image processing apparatus 5.

The image analysis unit 520 includes a distance calculation unit 521which processes the image signal group acquired from the imaging unit 10to calculate a distance to an object the image of which is picked. Thedistance calculation unit 521 calculates a distance from an arbitraryimage signal in an calculation range, i.e., an arbitrary pixel of animage pickup element to the object. The image analysis unit 520 formsdistance information in which a distance to the object calculated by thedistance calculation unit 521 is caused to correspond to a position ofthe object in an image and outputs the distance information to thecontrol unit 530.

The control unit 530 includes a window switching unit 531. The windowswitching unit 531 has functions of a switching process unit whichacquires moving information from the probing unit 560, selects a windowmatched with the moving information from pieces of window informationstored by window information 551, and outputs designation informationwhich switches the window set by the distance calculation unit 521 tothe selected window to the image analysis unit 520 together with thewindow information. The window information is information related to asize, a shape, and the like of the window.

The storage unit 550 stores the window information 551 in which piecesof window information selected by the window switching unit 531, probedinformation 552 probed by the probing unit 560, the image information553 picked by the imaging unit 10, and distance information 554calculated and formed by the distance calculation unit 521.

The probing unit 560 includes a speed sensor 561 which detects a movingspeed of the given vehicle and outputs the speed information detected bythe speed sensor 561 to the control unit 530 as moving information ofthe given vehicle. the speed sensor 561 may be a sensor which observes aroad surface on which the given vehicle runs from the given vehicle tocalculate a speed, a sensor which detects an acceleration to calculate aspeed, or the like.

In embodiment 5, the probing unit 560 constitutes at least part of amoving information detection unit 5600 which detects the movinginformation including the speed of the mobile object. The windowswitching unit 531 constitutes at least a part of a processing contentsetting unit 5310 which sets contents of a process to be performed in animage based on the moving information detected by the moving informationdetection unit 5600. Furthermore, the distance calculation unit 521constitutes at least a part of a processing calculation unit 5210 whichperforms processing calculation corresponding to the contents of theprocess set by the processing content setting unit 320.

Processes executed by the image processing apparatus 5 will be describedbelow with reference to the flow chart in FIG. 23. FIG. 23 is a flowchart showing procedures until the image processing apparatus 5 outputsdistance information corresponding to a picked image.

As shown in FIG. 23, the speed sensor 561 performs a speed probingprocess which detects a moving speed of the given vehicle and outputsthe detected speed to the control unit 530 as probing information (stepS501). The window switching unit 531 performs a window switching processwhich selects a window matched with the probing information acquiredfrom the probing unit 560 from the window information 551 and designatesthe image analysis unit 520 to switch a window to the selected window(step S503). In step S503, the window switching unit 531 outputsdesignation information which switches a window to the selected windowis output to the image analysis unit 520 together the windowinformation.

The imaging unit 10 performs an image pickup process which picks apredetermined field of view and outputs a generated image signal groupto the image analysis unit 520 as image information (step S505). Thedistance calculation unit 521 performs a distance calculation processwhich calculates a distance to an object on the basis of the imagesignal group corresponding to the window designated by the windowswitching unit 531, forms distance information in which the calculateddistance is caused to correspond to a position of the object on theimage, and outputs the distance information to the control unit 530(step S507). The control unit 530 outputs the distance information andpredetermined processing information based on the distance informationto an output unit 540 (step S509) to end a series of processes. Thecontrol unit 530 stores the probed information 552, the imageinformation 553, and the distance information 554 serving as informationgenerated in the processing steps as needed to the storage unit 550.

The series of processes are continuously repeated unless a predeterminedprocess end or a designation for interruption is received from apassenger or the like of the given vehicle. The series of processes aresequentially executed in the above description. However, it ispreferable that processes of independent processors are executed inparallel to each other to increase the speed of a processing cycle.Furthermore, a subsequent speed state may be predicted on the basis oftime-series speed information stored in the probed information 552, andthe speed probing process may be arbitrarily skipped to increase thespeed of the processing cycle. the image pickup process in step S505 maybe performed immediately before step S501 or step S503.

The window switching process in step S503 shown in FIG. 23 will bedescribed below. FIG. 24 is a flow chart showing procedures of thewindow switching process. As shown in FIG. 24, the window switching unit531 acquires the probing information from the probing unit 560 (stepS522) and determines whether a speed of the given vehicle is low or high(step S524). When the window switching unit 531 determines that thespeed is low (step S524: low speed), the window switching unit 531selects a low-speed window from the window information 551 (step S526),outputs designation information for switching a window to the selectedwindow to the image analysis unit 520 together with the windowinformation (step S528), and returns the operation to step S503). On theother hand, when the window switching unit 531 determines that the speedis high (step S524: high speed), the window switching unit 531 selects ahigh-speed window from the window information 551 (step S530), outputsdesignation information for switching a window to the selected window tothe image analysis unit 520 together with the window information (stepS523), and returns the operation to step S503.

In the flow chart shown in FIG. 24, the windows of two types, i.e., thelow-speed window or the high-speed window are selected on the basis ofthe speed of the given vehicle. However, a large number of speed statesmay be further determined to make it possible to select windows havingvarious sizes. Windows having different sizes may be substantiallydesigned to be selected to an arbitrary speed.

An example of a window selected by the window switching unit 531 will bedescribed below. FIG. 25 is a diagram showing an example of thelow-speed windows. A low-speed window 551 a shown in FIG. 25 correspondsto an almost entire region of an image pickup range 17 corresponding toa field of image pickup view of the imaging unit 10. A low-speed window515 a captures an object located in a range having a short distance or along distance to the given vehicle in the field of image pickup view onan image.

On the other hand, FIG. 26 is a diagram showing an example of thehigh-speed window. A high-speed window 551 b is smaller than thelow-speed window 551 a and corresponds to a partial region of the centerportion of the image pickup range 17. The high-speed window 551 b canselectively capture an object located at a long distance from the givenvehicle in the field of image pickup view on the image. As a result,long-distance image information which is important in running at a highspeed can be extracted.

FIG. 27 is a diagram obtained by overlapping the low-speed window 551 aon an image 18 a. The low-speed window 551 a, as shown in FIG. 27, avehicle C1 running at a position close to the given vehicle is capturedtogether with a background and targets the vehicle C1 for distancecalculation.

On the other hand, FIG. 28 is a diagram obtained by overlapping thelow-speed window 551 a on an image 18 b. The high-speed window 551 b, asshown in FIG. 28, captures a vehicle C2 running at a position having along distance from the given vehicle together with a background andtargets the vehicle C2 for distance calculation. The images 18 a and 18b are images picked by any one of the right camera 11 a and the leftcamera 11 b of the imaging unit 10.

An example of distance information calculated and formed by the distancecalculation unit 521 will be described below. FIG. 29 is a diagramshowing distance information 554 a formed by the distance calculationunit 21 on the basis of the image 18 a and the low-speed window 551 ashown in FIG. 27. In the distance information 554 a, a calculationresult 521 a represents a result of distance calculation in thelow-speed window 551 a. The calculation result 521 a, as shown in FIG.29, is expressed as a numerical value of a result obtained such that thedistance calculation unit 521 divides a region in the low-speed window551 a into a small regions arranged in a predetermined matrix andcalculates an average of distances to the object of the divided smallregions. In the calculation result 521 a, a result included in a regionEsa expresses a distance corresponding to sky which is the background 18a, and a result included in a region Ec1 expresses a distancecorresponding to the vehicle C1. Results except for the resultsdescribed above expresses a distance corresponding to a road surface.

On the other hand, FIG. 30 is a diagram showing a distance information554 b formed by the distance calculation unit 521 on the basis of theimage 18 b and the high-speed window 551 b shown in FIG. 28. In thedistance information 554 b, a calculation result 521 b expresses aresult of distance calculation in the high-speed window 551 b. Thecalculation result 521 b, like the calculation result 521 a, expressesan average of distances to the object calculated for the small regions.In the calculation result 521 b, a result included in a region Esbcorresponds to sky, a result included in a region Ec2 corresponds to thevehicle C2, and a result except for the results described abovecorresponds to a road surface. Numerical values expressed by thecalculation results 521 a and 521 b are numerical values obtained byexpressing distances by a predetermined unit system, for example, anumerical value expressed by the metric system.

The image processing apparatus 5 according to the fifth embodimentdescribed above selects an image processing range on the basis of amoving speed of a given vehicle and performs distance calculation on thebasis of an image signal group corresponding to the selected imageprocessing range. For this reason, in comparison with a conventionalimage processing apparatus which performs distance calculation to allimage signals of an image signal group, a load of a process for distancecalculation can be reduced, and time required for distance calculationcan be shortened. As a result, the image processing apparatus 5 canshorten processing time required until the distance information isoutput after an image is acquired and can output the distanceinformation at a high speed.

In the fifth embodiment, a moving speed, a moving direction, or positioninformation are detected as moving information of a given vehicle toselect a window. However, a moving acceleration, a moving orientation,and the like may be detected to select a window. A direction or a sightline of driver's face of a given vehicle may be detected to select awindow.

A sixth embodiment of the present invention will be described below. Inthe fifth embodiment described above, the window switching unit 531selects a window on the basis of a speed detected by the speed sensor561. However, in the sixth embodiment, a moving direction of a givenvehicle is detected to select a window.

FIG. 31 is a block diagram showing a configuration of an imageprocessing apparatus according to the sixth embodiment of the presentinvention. An image processing apparatus 6 shown in FIG. 31 includes aprobing unit 660 (at least a part of a moving information detection unit6600) in place of the probing unit 560 of the image processing apparatus5 according to the fifth embodiment described above. The probing unit660 includes a steering angle sensor 661. The image processing apparatus6 includes a control unit 630 in place of the control unit 530. Thecontrol unit 630 includes a window switching unit 631 (at least a partof a processing content setting unit 6310). The image processingapparatus 6 includes a storage unit 650 in place of the storage unit550. The storage unit 650 stores window information 651 in which piecesof window information selected by the window switching unit 631 arestored and probing information 652 probed by the probing unit 660, andstores the image information 553 and the distance information 554 as inthe storage unit 550. The remaining configuration is the same as that inthe fifth embodiment. The same reference numerals as in the fifthembodiment denote the same parts in the sixth embodiment.

The steering angle sensor 661 is a sensor which detects a steering angleserving as right and left rotating angles of front wheels as a movingdirection, for example a sensor which detects a steering angle on thebasis of a rotating angle and a rotating direction of a steering wheel.The probing unit 660 outputs the steering angle detected by the steeringangle sensor 661 to the control unit 630 as moving information of agiven vehicle. The steering angle sensor 661 may be a sensor whichdetects a moving direction by using a gyroscope, a sensor which detectsa moving direction by a blinking state of a direction indicator, or thelike.

Processes executed by the image processing apparatus 6 will be describedbelow. FIG. 32 is a flow chart showing procedures performed until theimage processing apparatus 6 outputs distance information correspondingto a picked image.

As shown in FIG. 32, the steering angle sensor 661 performs a steeringangle probing process which detects a steering angle of a given vehicleand outputs the detected steering angle to the control unit 630 asprobing information (step S601). The window switching unit 631 performsa window switching process which selects a window matched with probinginformation acquired from the probing unit 660 from the windowinformation 651 and designates the image analysis unit 520 to switch awindow to the selected window (step S603). In this case, the windowswitching unit 631 outputs designation information for switching awindow to the selected window to the image analysis unit 520 togetherwith the window information.

Thereafter, the image processing apparatus 6, as shown in FIG. 32,executes an image pickup process (step S605), a distance calculationprocess (step S607), and a distance information output process (stepS609) as in step S505, step S507, and step S509 shown in FIG. 23,respectively. The processes of the image processing apparatus 6 aredifferent from the processes of the image processing apparatus 5 in thata window used in the distance calculation process in step S607 is awindow corresponding to a steering angle of the given vehicle and thatthe steering angle detected by the steering angle sensor 661 is storedas the probing information 652.

In the image processing apparatus 6, as in the image processingapparatus 5, processes of independent processors are preferably executedin parallel to each other to increase the speed of a processing cycle.Furthermore, a subsequent state of a steering angle may be predicted onthe basis of time-series steering angle information stored by theprobing information 652, and the steering angle probing process may bearbitrarily skipped to increase the speed of the processing cycle. Theimage pickup process in step S605 may be performed immediately beforestep S601 or step S603.

The window switching process in step S603 shown in FIG. 32 will bedescribed below. FIG. 33 is a flow chart showing procedures of thewindow switching process. As shown in FIG. 33, the window switching unit631 acquires the probing information from the probing unit 660 (stepS622), and determines whether a direction of movement of the givenvehicle is right, left, or straightforward (step S624). When the windowswitching unit 631 determines that the direction is right (step S624:right), the window switching unit 631 selects a right window from thewindow information 651 (step S626), outputs designation information forswitching a window to the selected window to the image analysis unit 520together with the window information (step S628), and returns theoperation to step S603.

On the other hand, when the window switching unit 631 determines thatthe direction is left (step S624: left), the window switching unit 631selects a left window (step S630) and outputs designation informationfor switching a window to the selected window to the image analysis unit520 together with the window information (step S632), and returns theoperation to step S603. On the other hand, when the window switchingunit 631 determines that the direction is straightforward (step S624:straightforward), the window switching unit 631 selects astraightforward window (step S634), and outputs designation informationfor switching a window to the selected window to the image analysis unit520 together with the window information (step S636), and returns theoperation to step S603).

In the flow chart shown in FIG. 33, the windows of three types, i.e.,the right window, the left window, and the straightforward window areselected on the basis of a steering angle of the given vehicle. However,a large number of steering angle states may be further determined tomake it possible to select windows having various positions and varioussizes, or different windows may be able to be substantially selected toan arbitrary steering angle.

An example of a window selected by the window switching unit 631 will bedescribed below. FIG. 34 is a diagram showing an example of the rightwindow. A right window 651 a shown in FIG. 34 corresponds to an almostentire area of the right half of the image pickup range 17. The rightwindow 651 a can selectively capture an object located on the right sidein a field of image pickup view on the image. As a result, imageinformation on a forward right side which is most important when theroute is turned to the right can be extracted.

On the other hand, FIG. 35 is a diagram showing an example of a leftwindow. A left window 651 b corresponds to an almost entire area of theleft half of the image pickup range 17. The left window 651 bselectively captures an object located on the left side in the field ofimage pickup view on the image to make it possible to extract the imageinformation on the forward left side.

On the other hand, FIG. 36 is a diagram showing an example of thestraightforward window. A straightforward window 651 c corresponds to analmost entire area of the image pickup range 17. The straightforwardwindow 651 c captures an object located in an almost entire area in thefield of image pickup view on the image.

The image processing apparatus 6 according to the sixth embodimentdescribed above can select an image processing range on the basis of asteering angle of a given vehicle and perform distance calculation onthe basis of an image signal group corresponding to the selected imageprocessing range. For this reason, in comparison with a conventionalimage processing apparatus which performs distance calculation to allimage signals of an image signal group, a load of a process for distancecalculation can be reduced, and time required for distance calculationcan be shortened. As a result, the image processing apparatus 6 canshorten processing time required until the distance information isoutput after an image is acquired and can output the distanceinformation at a high speed.

A seventh embodiment of the present invention will be described below.In the fifth and the sixth embodiments described above, the windowswitching unit selects a window on the basis of a speed or a steeringangle detected by the probing unit. However, in the seventh embodiment,a position of a given vehicle is detected, map information is referredto on the basis of the detected position, and a window is selected onthe basis of the referred map information.

FIG. 37 is a block diagram showing a configuration of an imageprocessing apparatus according to the seventh embodiment of the presentinvention. An image processing apparatus 7 shown in FIG. 37 includes anexternal communication unit 770 (at least a part of a moving informationdetection unit 7700) in place of the probing unit 560 of the imageprocessing apparatus 5 according to the fifth embodiment describedabove. The image processing apparatus 7 includes a control unit 730 inplace of the control unit 530. The control unit 730 includes a windowswitching unit 731 (at least a part of a processing content setting unit7310). Furthermore, the image processing apparatus 7 includes a storageunit 750 in place of the storage unit 550. The storage unit 750 storeswindow information 751 which pieces of window information selected bythe window switching unit 731, position information 752 received by theexternal communication unit 770, and map information 753 to which thewindow switching unit 731 refers to on the basis of the positioninformation 752. As in the storage unit 550, the storage unit 750 storesthe image information 553 and the distance information 554. Theremaining configuration is the same as that in the fifth embodiment. Thesame reference numerals as in the fifth embodiment denote the same partsin the seventh embodiment.

The external communication unit 770 is a means for detecting a positionof a given vehicle, for example a communication means using a GPS(Global Positioning System). The external communication unit 770 outputsa detected position to the control unit 730 as moving information of thegiven vehicle. The external communication unit 770 may detect theposition of the given vehicle and acquire map information near thedetected position. As the external communication unit 770, in place ofthe communication means, a sensor or the like which detects a positionby applying a gyroscope or a speedometer.

Processes executed by the image processing apparatus 7 will be describedbelow. FIG. 38 is a flow chart showing procedures performed until theimage processing apparatus 7 outputs distance information correspondingto a picked image.

As shown in FIG. 38, the external communication unit 770 performsposition information reception in which position information of a givenvehicle is received and output to the control unit 730 as movinginformation (step S701). The window switching unit 731 performs a windowswitching process which refers to the map information 753 on the basisof the position information acquired from the external communicationunit 770, selects a window matched with the referred map informationfrom the window information 751, and designates the image analysis unit520 to switch a window to the selected window (step S703). In this case,the window switching unit 731 outputs designation information whichswitches a window to the selected window to the image analysis unit 520together with the window information.

Thereafter, the image processing apparatus 7, as shown in FIG. 38,executes an image pickup process (step S705), a distance calculationprocess (step S707), and a distance information output process (stepS709) as in step S505, step S507, and step S509 shown in FIG. 23,respectively. The processes of the image processing apparatus 7 aredifferent from the processes of the image processing apparatus 5 in thata window used in the distance calculation process in step S707 is awindow corresponding to position information and map information andthat the position detected by the external communication unit 770 isstored as the position information 752.

In the image processing apparatus 7, as in the image processingapparatus 5, processes of independent processors are preferably executedin parallel to each other to increase the speed of a processing cycle.Furthermore, a subsequent position may be predicted on the basis oftime-series position information or the like stored by the positioninformation 752, and the steering angle probing process may bearbitrarily skipped to increase the speed of the processing cycle. Theimage pickup process in step S705 may be performed immediately beforestep S701 or step S703.

An example of the window switching process in step S703 shown in FIG. 38will be described below. FIG. 39 is a flow chart showing an example ofprocedures of the window switching process. As shown in FIG. 39, thewindow switching unit 731 acquires the position information from theexternal communication unit 770 (step S722), refers to the mapinformation 753 on the basis of the acquired position information (stepS724), and determines whether the given vehicle runs on a freeway (stepS726).

The window switching unit 731 selects a freeway window from the windowinformation 751 (step S728) when the window switching unit 731determines that the given vehicle runs on the freeway (step S726: Yes),outputs designation information which switches a window to the selectedwindow to the image analysis unit 520 together with the windowinformation (step S730), and returns the operation to step S703.

On the other hand, when the window switching unit 731 determines thatthe given vehicle does not runs on the freeway (step S726: No), thewindow switching unit 731 selects a standard window (step S732, outputsdesignation information which switches a window to the selected windowto the image analysis unit 520 together with the window information(step S734), and returns the operation to step S703.

An example of the window selected by the window switching unit 731 willbe described below. FIG. 40 is a diagram showing an example of a freewaywindow. A freeway window 751 a shown in FIG. 40 is a window constitutedby a plurality of small windows arranged at a center portion of theimage pickup range 17. The freeway window 751 a can selectively capturean object located at a center portion of a field of image pickup view onan image, and can further extracts image information from the imageinformation in the captured range at a predetermined rate. As a result,the freeway window 751 a can extract image information on a road whichis important when a vehicle runs on a freeway with an appropriate imageresolution.

On the other hand, FIG. 41 is a diagram showing an example of thestandard window. A standard window 751 b corresponds to an almost entireregion of the image pickup range 17. The standard window 751 b capturesan object which is not only on a road but also in the almost entireregion in the field of image pickup view on an image.

The various patterns of the freeway window can be set. For example, as aslit pattern of the freeway window 751 a shown in FIG. 40, a slitpattern including a large number of slits each having a further narrowwidth may be used, or a dot pattern obtained by arranging a plurality ofsmall square windows at random may be used. Other various patterns canbe applied. The shape of the dot of the dot pattern is not limited to asquare but also an arbitrary polygon, a circle, or an ellipse.

Another example of the window switching process in step S703 shown inFIG. 38 will be described below. FIG. 42 is a flow chart showingprocedures of the window switching process including a process ofselecting a window on the basis of an inclination of a road surfacegradient of a road on which a vehicle runs. As shown in FIG. 42, thewindow switching unit 731 acquires position information from theexternal communication unit 770 (step S742), refers to the mapinformation 753 on the basis of the acquired position information (stepS744), and determines whether a road surface on which the given vehicleruns is a convex surface, concave surface, or a flat surface (stepS746).

When the window switching unit 731 determines that the road surface onwhich the given vehicle runs is a concave surface (step S746: concavesurface), the window switching unit 731 selects a concave surface windowfrom a window information 351 (step S748), outputs designationinformation which switches a window to the selected window to the imageanalysis unit 520 together with the window information (step S750), andreturns the operation to step S703. On the other hand, when the windowswitching unit 731 determines that the road surface is a convex surface(step S746: convex surface), the window switching unit 731 selects aconvex surface window (step S752), outputs designation information whichswitches a window to the selected window to the image analysis unit 520together with the window information (step S754), and returns theoperation to step S703.

On the other hand, the window switching unit 731 determines that theroad surface is a flat surface (step S746: flat surface), the windowswitching unit 731 selects a standard window (step S756), outputsdesignation information which switches a window to the selected windowto the image analysis unit 520 together with the window information(step S758), and returns the operation to step S703.

The concave road surface means that a road surface gradient tends toincrease at an elevation angle direction. For example, a road surfacewhere a flat road surface changes into an ascending slope or a roadsurface where a descending slope changes into a flat road surface isapplied. On the other hand, the convex road surface means that a roadsurface gradient tends to increase at a depression angle direction. Forexample, as the convex road surface, a road surface where a flat roadsurface changes into a descending slope or a road surface where anascending slope changes into a flat road surface is applied.

An example of a window selected by the window switching unit 731 on thebasis of an inclination of a road surface gradient will be describedbelow. FIG. 43 is a diagram showing an example of the concave surfacewindow. The concave surface window 751 c shown in FIG. 43 corresponds toan almost entire area of the upper half of the image pickup range 17.The concave surface window 751 c can selectively capture an objectlocated on the upper side in a field of image pickup view on the image.As a result, image information on a forward upper side which isimportant when the vehicle runs on a road surface having a concavesurface can be extracted.

On the other hand, FIG. 44 is a diagram showing an example of a convexsurface window. A convex surface window 751 d shown in FIG. 44corresponds to an almost entire area of the lower half of the imagepickup range 17. The convex surface window 751 d selectively captures anobject located on the lower side in the field of image pickup view onthe image to make it possible to extract image information on theforward lower side. The standard window selected when the road surfaceis flat surface is the same as the standard window 751 b shown in FIG.41.

In the flow charts shown in FIGS. 39 and 42, a window is selected as mapinformation on the basis of a type of a road on which a given vehicleruns or an inclination of a road surface gradient. For example, variouswindows may be selected on the basis of various pieces of mapinformation such that information representing whether a road or a laneon which a vehicle runs is large or small, information representingwhether a present position is in a tunnel or the like, and informationrepresenting whether a present position is in an urban area.

The image processing apparatus 7 according to the seventh embodimentdescribed above selects an image processing range on the basis ofposition information of the given vehicle and map information which isreferred to on the basis of the position information, and distancecalculation is performed on an image signal group corresponding to theselected image processing range. For this reason, in comparison with aconventional image processing apparatus which performs distancecalculation to all image signals of an image signal group, a load of aprocess for distance calculation can be reduced, and time required fordistance calculation can be shortened. As a result, the image processingapparatus 7 can shorten processing time required until the distanceinformation is output after an image is acquired and can output thedistance information at a high speed.

An eighth embodiment of the present invention will be described below.An image processing apparatus according to the eighth embodimentincludes all the speed sensor 561, the steering angle sensor 661, andthe external communication unit 770 which are included in the imageprocessing apparatuses 5, 6, and 7 according to the fifth to the seventhembodiments described above, multilaterally detects moving informationof the given vehicle, and selects a window matched with the detectedmoving information.

FIG. 45 is a block diagram showing a configuration of the imageprocessing apparatus according to the eighth embodiment of the presentinvention. An image processing apparatus 8 shown in FIG. 45 furtherincludes a probing unit 860 in the image processing apparatus 7according to the seventh embodiment described above. The probing unit860 includes a speed sensor 561 and a steering angle sensor 661. Theimage processing apparatus 8 includes a control unit 830 in place of thecontrol unit 730. The control unit 830 includes a window switching unit831 (at least a part of a processing content setting unit 8310).Furthermore, the image processing apparatus 8 includes a storage unit850 in place of the storage unit 750. The storage unit 850 stores windowinformation 851 in which pieces of window information selected by thewindow switching unit 831 are stored and probing information 852 probedby the probing unit 860, and stores, as in the storage unit 750, theimage information 553, the distance information 554, the positioninformation 752, and the map information 753. The remainingconfiguration is the same as that in the seventh embodiment. The samereference numerals as in the seventh embodiment denote the same parts inthe eighth embodiment.

In the image processing apparatus 8, the window switching unit 831independently or totally determines a speed detected by the speed sensor561, a steering angle detected by the steering angle sensor 661, and mapinformation based on position information received by the externalcommunication unit 770, selects a window matched with the determinationresult from the window information 851, and outputs designationinformation which switches a window to the selected window to the imageanalysis unit 520 together with the window information. In this sense,in the eighth embodiment, the probing unit 860 and the externalcommunication unit 770 constitute at least a part of a movinginformation detection unit 8700 as a whole.

Thereafter, the distance calculation unit 521, as in the fifth to theseventh embodiments described above, calculates a distance to an objecton the basis of an image signal group corresponding to the selectedwindow, forms distance information, and outputs the distanceinformation. The window switching unit 831 may recognize at least oneavailable combination of various pieces of moving information acquiredfrom the probing unit 860 or the external communication unit 770 as amode and determine a state of the given vehicle with respect to a modedesignated by a passenger or the like of the given vehicle. The windowswitching unit 831 may predict a subsequent state of the given vehicleon the basis of time-series moving information stored by the probinginformation 852 and the position information 752.

The image processing apparatus 8 according to the eighth embodimentdescribed above selects an image processing range on the basis of aresult obtained by independently or totally determining various piecesof moving information of the given vehicle, and performs distancecalculation on the basis of the image signal group corresponding to theselected image processing range. For this reason, in comparison with aconventional image processing apparatus which performs distancecalculation to all image signals of an image signal group, a load of aprocess for distance calculation can be reduced, and time required fordistance calculation can be shortened. As a result, the image processingapparatus 8 can shorten processing time required until the distanceinformation is output after an image is acquired and can output thedistance information at a high speed.

A ninth embodiment of the present invention will be described below. Inthe fifth to the eighth embodiments, a distance to an object the imageof which is picked is detected by processing an image signal group fromthe imaging unit 10. However, in the fifth embodiment, a distance to anobject located in the field of image pickup view is detected by a radar.

FIG. 46 is a block diagram showing a configuration of an imageprocessing apparatus according to the ninth embodiment of the presentinvention. An image processing apparatus 9 shown in FIG. 46 includes inaddition to the image processing apparatus 8 according to the eighthembodiment described above, a radar 980 and a control unit 930 having afunction of controlling the radar 980 in place of the control unit 830.The remaining configuration is the same as that in the eighthembodiment. The same reference numerals as in the eighth embodimentdenote the same parts in the ninth embodiment. In the ninth embodiment,the external communication unit 770, the probing unit 860, and the radar980 constitute a part of a moving information detection unit 9870 as awhole.

The radar 980 transmits a predetermined outgoing wave, receives areflected wave obtained by reflecting the outgoing wave by an objectsurface, and detects a receiving state and a distance from the radar 980to the object which reflects the outgoing wave and a direction in whichthe object is located on the basis of the receiving state. The radar 980detects the distance and the direction of the object which reflects theoutgoing wave on the basis of an outgoing angle of the outgoing wave, anincident angle of the reflected wave, a receiving intensity of thereflected wave, time from when the outgoing wave is emitted to when thereflected wave is received, and a change in frequency in the outgoingwave and the reflected wave. The radar 980 outputs a distance to anobject located in a field of image pickup view of the imaging unit 10 tothe control unit 930 together with the direction in which the object islocated. The radar 980 transmits, for example, a laser beam, an infraredray, a millimeter wave, a microwave, an ultrasonic wave, or the like asa transmitter wave.

In the image processing apparatus 9 according to the ninth embodiment, adistance is detected by the radar 980 in place of calculation of adistance by processing image information obtained from the imaging unit10. For this reason, distance information can be acquired at a higherspeed and a higher accuracy.

In the image processing apparatus 9, matching between a positionalrelationship in the image signal group picked by the imaging unit 10 anda positional relationship in the detection range of the radar 980 isobtained as follows, and then the processes are performed. For example,the image processing apparatus 9 performs an image pickup process by theimaging unit 10 and a detection process by the radar 980 to an objecthaving a known shape. A position of the known object processed by theimaging unit 10 and the radar 980 is obtained. Thereafter, the imageprocessing apparatus 9 obtains a relationship between positions of theknown object processed by the imaging unit 10 and the radar 980 by usingthe method of least squares and matches the positional relationship inthe image signal group picked by the imaging unit 10 with the positionalrelationship in the detection range of the radar 980.

In the image processing apparatus 9, even though an image pickuporiginal point of the imaging unit 10 and a detection original point ofthe radar 980 are different from each other, if distances from the imagepickup point and the detection point to the image processing apparatus 9are sufficiently long, it can be understood that an image pickuporiginal point and a detection original point are almost overlapped.Furthermore, when matching between the positional relationship in theimage signal group picked by the imaging unit 10 and the positionalrelationship in the detection range of the radar 980 is accuratelyperformed, the difference between the image pickup original point andthe detection original point can be corrected by geometric conversion.

In the image processing apparatus 9, radar detection points of the radar980 are preferably set to be located at predetermined intervals in apixel row in which image signals of the image signal group picked by theimaging unit 10. When the radar detection points are not set asdescribed above, interpolation points of the radar detection points areobtained in the same pixel row as the row of the image signals by usingprimary interpolation or the like on the basis of a plurality of radardetection points located near the image signals, and a detection processmay be performed by the interpolation points.

FIG. 47 is a block diagram showing a configuration of an imageprocessing apparatus according to a tenth embodiment of the presentinvention. An image processing apparatus 101 shown in FIG. 47 is mountedon a mobile object, in particular, a vehicle such as a four-wheelautomobile, and is constituted by an imaging unit 10 which picks apredetermined field of view, an image analysis unit 1020 which analyzesan image generated by the imaging unit 10, a control unit 1030 whichperforms operation control of the image processing apparatus 101, anoutput unit 40 which outputs information such as an image or letters ona display, a storage unit 1050 which stores various data, a speedsensor, and the like. The image processing apparatus 101 includes aspeed detection unit 1060 which detects a speed of a given vehicle.

The image analysis unit 1020 has a distance information generating unit1021 which calculates a distance to an object the image of which is tobe picked and which is included in a region of a field of view the imageof which is picked by the imaging unit 10 and an image processing unit1022 which performs image processing corresponding to the distanceinformation and a speed of a given vehicle detected by the speeddetection unit 1060. As is apparent from this explanation, the distanceinformation generating unit 1021 has a function of a distancecalculation unit.

The control unit 1030 has a processing selection unit 1031 which selectsan image processing method actually performed by the image processingunit 1022 from a plurality of image processing methods.

The storage unit 1050 stores and memorizes image data 1051 picked by theimaging unit 10, distance information 1052 of constituent points (pixelpoints) of the image data 1051, an image processing method 1053 to beselected by the processing selection unit 1031, a template 1054 whichexpresses patterns of various objects (vehicle, person, road surface,white line, indicator, and the like) used in object recognition in animage in units of pixel points, and zone dividing information 1055obtained by dividing a range of an available speed of a running vehicleand a range of distance in which image pickup can be performed by theimaging unit 10 into pluralities of zones, respectively.

In the tenth embodiment, the speed detection unit 1060 constitutes atleast a part of a moving information detection unit 1160 which detectsmoving information including a speed of a mobile object. The processingselection unit 1031 constitutes at least a part of a processing contentsetting unit 1131 which sets contents of processing to be performed inan image based on moving information detected by the moving informationdetection unit 1160. Furthermore, the image processing unit 1022constitutes at least a part of a processing calculation unit 1122 whichperforms processing calculation corresponding to the contents of theprocessing set by the processing content setting unit 1131.

An image processing method executed by the image processing apparatus101 having the above configuration will be described below in detailwith reference to a flow chart shown in FIG. 48. The imaging unit 10performs an image pickup process which picks a predetermined field ofview to generate an image (step S1001).

Thereafter, the distance information generating unit 1021 calculates adistance to an object the image of which is to be picked and which isincluded in the image of which is picked by each image of pixel points(step S1003). The distance information generating unit 1021 calculatescoordinates of all pixel points or some pixel points in the field ofview picked by using right and left camera coordinate systems.Subsequently, the distance information generating unit 1021 calculates adistance R from the forehead surface of the vehicle to a point the imageis picked by using the calculated coordinates (x, y, z) of the pixelpoints. In this case, a position of the forehead surface of the vehiclein the camera coordinate system must be predicted in advance.Thereafter, the distance information generating unit 1021 stores thecalculated coordinates (x, y, z) of all the pixel points or some pixelpoints and the distance R in the storage unit 1050.

The distance information including the distance calculated in step S1003may be superposed on the image generated in step S1001 to generate adistance image. FIG. 49 is a diagram showing a display output example ofthe distance image in the output unit 40. A distance image 301 shown inFIG. 49 expresses a distance from the imaging unit 10 in a grayscale. Aportion corresponding to a distant place is displayed with a higherdensity.

In parallel to the processes (step S1001 and S1003) in the imaging unit10 and the distance information generating unit 1021, the speeddetection unit 1060 detects a speed of a given vehicle (step S1005).

The processing selection unit 1031 in the control unit 1030, on thebasis of the distance calculation result in step S1003 and the speeddetection result in step S1005, selects an image processing methodexecuted to the points in the image by the image processing unit 1022depending the distance calculated in step S1003 from the imageprocessing methods 1053 stored in the storage unit 1050 (step S1007).Thereafter, the image processing unit 1022 performs image processingaccording to the image processing method selected by the processingselection unit 1031 (step S1009). At this time, the image processingunit 1022 reads the image processing method selected by the processingselection unit 1031 from the storage unit 1050, and performs the imageprocessing based on the read image processing method.

FIG. 50 is a diagram showing an example of the image processing methodselected by the processing selection unit 1031. In a corresponding table71 shown in FIG. 50, each of a distance to a region to be processed anda speed of a given vehicle is divided into two zones, and an imageprocessing method to be selected depending on combinations of zones towhich the distance and the speed belong. In this selection, theprocessing selection unit 1031 determines zones to which a calculateddistance and a detected speed belong with reference to the zone dividinginformation 1055 stored in the storage unit 1050, and collates thedetermination result with the image processing method 1053 in thestorage unit 1050. A control signal related to the image processingmethod to be performed by the image processing unit 1022 is transmittedto the image processing unit 1022.

Processing contents shown in FIG. 50 will be concretely described below.In the tenth embodiment, a predetermined object is recognized in eachzone. In recognition of the object, patterns (vehicle, person, roadsurface, white line, indicator, and the like) stored in the template1054 of the storage unit 1050 with a pattern picked by the imaging unit10 in an image to examine correlations between the patterns (templatematching). When a corresponding pattern is detected as a result of thetemplate matching, processing depending on the detected object isperformed. Therefore, the image processing unit 1022 has a function ofan object detection unit which detects an object.

When the processing region belongs to a long-distance zone, and when avehicle moves at a speed belonging to a high-speed zone, vehicledetection is performed to the processing region. In the vehicledetection performed here, when a brake must be applied when a precedingvehicle comes close to the given vehicle by a predetermined distance,image processing is performed such that a driver is notified that thebrake must be applied. For this reason, when a pattern of the vehiclecorresponding to a size obtained when the preceding vehicle comes closeto the given vehicle by the predetermined distance is stored in thetemplate 1054, the vehicle is not detected when the preceding vehicleruns ahead of the pattern.

FIG. 51 is a diagram showing a display example of an image output fromthe output unit 40 when the vehicle is detected. In a display image 401shown in FIG. 51, since a preceding vehicle is detected, a messagerepresenting “Please apply brake” is output. In accordance with thedisplay contents, the same contents at those displayed by letters may beoutput as voice, or an appropriate warning sound may be made to callattention to a vehicle operator.

A case in which the processing region belongs to a short-distance zoneand a vehicle moves at a speed belonging to a high-speed zone will bedescribed below. In this case, white line detection is performed to thecorresponding processing region. On the other hand, when the processingregion belongs to a long-distance zone, and when the vehicle moves at aspeed belonging to a low-speed zone, white line detection is performedto the processing region. As concrete image processing for the whileline detection, for example, when a white line is detected, and when apattern of the detected white line is different from that in a normalrunning state, the driver may be notified that the pattern is differentfrom that in the normal running state. In this case, a lane dividingbelt (yellow line or the like) except for the white line may bedetected.

FIG. 52 is a diagram showing an example of a display image output by theoutput unit 40 in this case. In a display image 402 shown in FIG. 52,the image processing unit 1022 determines that a direction or a patternof a detected white line is normal with respect to a running directionof a given vehicle, notifies a driver that the direction or the patternis not normal, a measure (“you will be out of lane, go to the right”) tobe made by a vehicle operator. In this case, the same contents at thosedisplayed by letters may be output as voice, or an appropriate warningsound may be output.

When the processing region belongs to the long-distance zone, and when avehicle moves at a speed belonging to a low-speed zone, detection of avehicle or a person is performed to the processing region. At the sametime, an obstacle may be detected. When the obstacle is detected, amessage representing “Please avoid obstacle” or the like may be output.

The image processing method described above is only an example. Anotherimage processing method may be selected by the processing selection unit1031. For example, in FIGS. 51 and 52, the case in which only one imageprocessing is performed in an image is explained. However, a combinationof pixel points belonging to the same distance zone in the image can beset as one closed region, and image processing can be performed to eachof set closed regions. In this case, the image analysis unit 1020achieves a function of a closed region setting unit.

According to the tenth embodiment of the present invention describedabove, an image processing method depending on a distance to an objectthe image of which is to be picked and which is generated by a pickedimage and a speed of a given vehicle serving as a mobile object, and theselected image processing method is applied, so that various pieces ofinformation included in the image to be picked can be multilaterally andefficiently processed.

In the tenth embodiment, a combination of a speed zone and a distanceregion is not limited to the combination described above. FIG. 53 is anexplanatory diagram showing a case in which each of the speed zone andthe distance zone is divided into three zones, and a process to beexecuted is set depending on 9 combinations constituted by the dividedzones. In a corresponding table 81 shown in FIG. 53, when a distance tothe processing region belongs to the short-distance zone, and when aspeed belongs to the low-speed zone, a person is detected. When acombination of the distance zone and the speed zone is (intermediatedistance and low speed) or (short distance and intermediate speed),detection for a vehicle or a person is performed. When the combinationof the distance zone and the speed zone is (long distance and highspeed), detection for a vehicle is performed. More specifically, when(long distance and low speed), (long distance and intermediate speed),(intermediate distance and intermediate speed), (intermediate distanceand high speed), or (short distance and high speed), detection for awhite line is performed. In this manner, the speed zone and the distancezone may be further divided into small zones, and image processing maybe performed depending on combinations of the small zones.

The distance zone is divided on the basis of a braking distance at thattime, and an image processing method can also be determined on the basisof the divided distance zone. In this case, for example, theshort-distance zone falls within a braking distance at that time, themiddle-distance zone is equal to or larger than a braking distance atthat time and equal to or smaller than two times the braking distance,and the long-distance zone is equal to or larger than two times thebraking distance at that time. “Obstacle+white line” are detected in theshort-distance zone, “Person+vehicle+obstacle+white line” are detectedin the intermediate-distance zone, and “white line” is detected in thelong-distance zone.

Furthermore, in addition to indexes of a distance and a speed, personalinformation of a vehicle driver, for example, a driving experience, anage, a sex, an operation experience of a vehicle which is beingoperated, and the like are input in advance. When divisions of the zonesare changed depending on the pieces of personal information, imageprocessing can be more exactly performed depending on conditions of avehicle operator.

A selection method changing unit which changes a selection method forimage processing methods in the processing selection unit can alsofurther arranged in the image processing apparatus according to thetenth embodiment. In this manner, an optimum image processing method canbe selected and executed depending on various conditions in travelingstate of the vehicle.

Even in the tenth embodiment, in addition to various cameras, a distancemeasuring apparatus such as a radar can also be used. In this manner, aradar measurement value having an accuracy higher than that of a normalcamera can be used, and a resolving power of distance measurement pointscan be further improved.

As a method of image recognition in the tenth embodiment, in addition tothe template matching described above, an object recognition methodwhich is popularly used such as a region dividing method by edgeextraction and a statistic pattern recognition method based on clusteranalysis can be applied.

FIG. 54 is a block diagram showing a configuration of an imageprocessing apparatus 201 according to a modification of the tenthembodiment. The image processing apparatus 201 shown in FIG. 54includes, as an image analysis unit 2020, in addition to the distanceinformation generating unit 1021 and the image processing unit 1022, adistance image generating unit 2021 which generates a distance image, aclosed region setting unit 2022 which sets different closed regions to aplurality of distance zones, respectively, and an object detection unit2023 which detects a predetermined object for each of the closed regionsset by the closed region setting unit 2022. According to the imageprocessing apparatus 201 having the above configuration, the sameprocesses as those in the image processing apparatus 101 can be realizedfor the plurality of closed regions.

An eleventh embodiment of the present invention will be described below.In the first to the tenth embodiments described above, a stereo image ispicked by two cameras, i.e., the right camera 11 a and the left camera11 b. However, in the eleventh embodiment, a stereo image is picked byan image pickup element which has one pair of waveguide optical systemsand image pickup regions corresponding to the waveguide optical systemsand converts optical signals guided by the waveguide optical systemsinto electric signals in the image pickup regions, respectively.

FIG. 55 is a block diagram showing a partial configuration of an imageprocessing apparatus according to the eleventh embodiment of the presentinvention. An imaging unit 110 shown in FIG. 55 is an imaging unit whichis arranged in the image processing apparatus according to the eleventhembodiment in place of the imaging unit 10 in the image processingapparatus 1 or the like described above. The configuration of the imageprocessing apparatus except for that shown in FIG. 55 is the same asthat described in any one of the first to the tenth embodiments.

The imaging unit 110 includes a camera 111 serving as an image pickupapparatus having the same configuration and the same functions as thoseof the right camera 11 a and the left camera 11 b of the imaging unit10. The camera 111 includes a lens 112, an image pickup element 113, anA/D conversion unit 114, and a frame memory 115. Furthermore, theimaging unit 110 includes a stereo adapter 119 serving as one pair ofwaveguide optical systems constituted by mirrors 119 a to 119 d in frontof the camera 111. The stereo adapter 119, as shown in FIG. 55, includesthe mirrors 119 a and 119 b as a pair such that reflecting surfaces ofthe mirrors almost horizontally face each other, and includes themirrors 119 c and 119 d as another pair such that reflecting surfaces ofthe mirrors almost horizontally face each other. The stereo adapter 119includes the two pairs of mirror systems which are adjacent to eachother and symmetrical about an optical axis of the lens 112.

In the imaging unit 110, light from an object located in a field ofimage pickup view is received by the two right and left pairs of mirrorsystems of the stereo adapter 119, converged by the lens 112 serving asan image pickup optical system, and picked by the image pickup element113. At this time, as shown in FIG. 56, the image pickup element 113picks a right image 116 a passing through the mirror system of the rightpair of mirrors 119 a and 119 b and a left image 116 b passing throughthe mirror system of the left pair of mirrors 119 c and 119 d are pickedin image pickup regions which are horizontally shifted from each otherand which are not overlapped at all. A technique using the stereoadapter described above is disclosed in, for example, Patent Document 2.

In the imaging unit 110 according to the eleventh embodiment, since astereo image is picked by one camera having a stereo adapter, an imagingunit can be simplified and made compact in comparison with aconfiguration in which a stereo image is picked by two cameras,mechanical strength can be increased, and right and left images can bealways picked in a relatively stable state. Furthermore, since right andleft images are picked by using the common lens and the common imagepickup element, a fluctuation caused by individual differences can besuppressed, and trouble of calibration and cumbersome assembly such asalignment can be reduced.

As a configuration of the stereo adapter, in FIG. 55, plane mirrors arecombined to each other such that the plane mirrors almost horizontallyface each other. However, lens groups may be combined to each other toconstitute a stereo adapter, or reflecting mirrors such as a convexmirror and a concave mirror each having a curvature may be combined toeach other to constitute a stereo adapter. In place of a reflectingmirror, a reflecting surface may be obtained by a prism.

In embodiment 11, as shown in FIG. 56, the right and left images arepicked such that the images are not overlapped at all. However, theright and left images are picked such that the images are partially orentirely overlapped. For example, lights received on the right and theleft may be sequentially picked while being switched by a shutterarranged on a right-receiving unit, and right and left images pickedwith a small time difference may be used as a stereo image in imageprocessing.

Furthermore, in the configuration in the eleventh embodiment, the rightand left images are picked while being horizontally shifted from eachother. For example, the plane mirrors of the stereo adapter may bealmost orthogonally combined to each other, and right and left imagesmay be picked while being vertically shifted.

Up to now, the preferable embodiments of the present invention have beendescribed in detail. However, the present invention is not limited bythe first to the eleventh embodiments described above. For example, as astereo camera of an imaging unit, a stereo camera having a larger numberof views, for example, a three-view stereo camera or a 4-view stereocamera may be arranged. It is known that a stable processing resulthaving higher reliability is obtained in a three-dimensionalreconfiguration process or the like by using the 3-view or 4-view stereocamera (see Tomita Fumiaki, “Advanced Three-Dimensional Vision SystemVVV”, Journal of Information Processing Society of Japan “InformationProcessing”, Vol. 42, No. 4, pp. 370 to 375 (2001) or the like). Inparticular, it is known that, when a plurality of cameras are arrangedto have base lengths in two directions, a three-dimensionalreconfiguration can be obtained in a more complex scene. In addition,when a plurality of cameras are arranged in one base length direction, astereo camera of a so-called multi-base line type can be realized, and amore accurate stereo measurement can be achieved.

Furthermore, as the camera of the imaging unit, in place of a fly-eyestereo camera, a simple-eye camera may be used. In this case, athree-dimensional reconfiguration technique such as a shape from focusmethod, a shape from defocus method, a shape from motion method, or ashape from shading method is applied to make it possible to calculate adistance to an object in a field of image pickup view.

The shape from focus method is a method which calculates a distance froma focus position when the best focus is obtained. The shape from defocusmethod is a method which obtains a relative amount of blur is obtainedfrom a plurality of images having different focal distances and obtainsa distance on the basis of a correlation between the amount of blur andthe distance. The shape promotion method is a method which obtains adistance to an object on the basis of a moving track of a predeterminedcharacteristic point in a plurality of images which continues in termsof time. The shape from shading method is a method which calculates adistance to an object on the basis of the light and shade on an image, areflecting characteristic of a target object, and light sourceinformation.

Furthermore, in the explanation of the configuration of the imaging unit10 in the first to the tenth embodiments or the configuration of theimaging unit 110 in the eleventh embodiment, one pair of cameras orlight-receiving units of the stereo adapter are arranged to behorizontally aligned. However, the cameras or the light-receiving unitsmay be vertically aligned or may be aligned in an oblique direction.

In the first to the eleventh embodiments, the image processing apparatusmount on a vehicle such as a four-wheel vehicle is explained. However,the image processing apparatus can be mounted on another mobile object,for example, an electric wheelchair or the like. The image processingapparatus can be mounted on not only a vehicle but also a mobile objectsuch as a person or a robot. Furthermore, an entire image processingapparatus need not be mounted on a mobile object. For example, animaging unit and an output unit are mounted on a mobile object, and theother constituent parts are arranged out of the mobile object, so thatthe imaging units, the output units, and the parts may be connected toeach other by wireless communication.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. An image processing apparatus comprising: an imaging unit which ismounted on a mobile object and picks an image of a predetermined fieldof view to generate an image; a moving information detection unit whichdetects moving information including a speed of the mobile object; aprocessing content setting unit which sets contents of a process to beperformed in the image generated by the imaging unit, based on themoving information detected by the moving information detection unit;and a processing calculation unit which performs processing calculationaccording to the contents of the process set by the processing contentsetting unit.
 2. The image processing apparatus according to claim 1,further comprising: an arrival point prediction unit which predicts apoint at which the mobile object arrives a predetermined period of timeafter based on the moving information detected by the moving informationdetection unit, wherein the processing content setting unit includes acalculation range setting unit which sets a calculation range in which adistance to an object included in the field of view based on aprediction result by the arrival point prediction unit, and theprocessing calculation unit includes a distance calculation unit whichperforms distance calculation in the calculation range set by thecalculation range setting unit.
 3. The image processing apparatusaccording to claim 2, wherein the arrival point prediction unit predictsa point at which the mobile object arrives a predetermined period oftime after based on a function experientially obtained.
 4. The imageprocessing apparatus according to claim 2, wherein the arrival pointprediction unit corrects the prediction result based on the movinginformation.
 5. The image processing apparatus according to claim 3,wherein the calculation range prediction unit sets the calculation rangebased on the function experientially obtained.
 6. The image processingapparatus according to claim 2, wherein the moving information is aspeed of the mobile object, an acceleration of the mobile object, amoving direction of the mobile object, a moving orientation of themobile object, position information of the mobile object, a change rateof the moving information, or a combination thereof.
 7. The imageprocessing apparatus according to claim 2, wherein the imaging unitgenerates a first image signal group picked through a first optical pathand a second image signal group picked through a second optical path,and the distance calculation unit detects an image signal matched withan arbitrary image signal of the first image signal group from thesecond image signal group, and calculates a distance to the object basedon an amount of movement of the detected image signal from the arbitraryimage signal.
 8. The image processing apparatus according to claim 1,wherein the processing content setting unit includes a switching processunit which selects an image processing range to be calculated by theprocessing calculation unit from a plurality of image processing rangesincluded in the image, based on the moving information output from themoving information detection unit, and switches to the selected imageprocessing range and the processing calculation unit includes a distancecalculation unit which calculates a distance to an object included inthe field of view based on an image signal group corresponding to theimage processing range switched by the switching process unit.
 9. Theimage processing apparatus according to claim 8, wherein the movinginformation is at least one of a moving speed of the mobile object, amoving acceleration of the mobile object, a moving direction of themobile object, a moving orientation of the mobile object, and positioninformation of the mobile object.
 10. The image processing apparatusaccording to claim 8, wherein the imaging unit generates as the imagesignal group a first image signal group picked through a first opticalpath and a second image signal group picked through a second opticalpath, and the distance calculation unit detects an image signal matchedwith an arbitrary image signal of the first image signal group from thesecond image signal group, and calculates a distance to the object basedon an amount of movement of the detected image signal from the arbitraryimage signal.
 11. The image processing apparatus according to claim 1,further comprising: a distance calculation unit which calculates adistance from an image pickup position of the imaging unit to an objectthe image of which is to be picked, wherein the processing contentsetting unit includes a processing selection unit which selects an imageprocessing method from a plurality of image processing methods, based onthe distance from the image pickup position to the object the image ofwhich is to be picked obtained by the distance calculation unit and aspeed of the mobile object obtained by the moving information detectionunit, and the processing calculation unit includes an image processingunit which applies the image processing method selected by theprocessing selection unit to perform image processing to the image. 12.The image processing apparatus according to claim 11, furthercomprising: a zone dividing information storage unit which divides arange of an available speed of the mobile object into a plurality ofspeed zones to store the plurality of speed zones, and divides a rangeof a distance at which image pickup can be performed by the imagingunit, into a plurality of distance zones to store the plurality ofdistance zones, wherein the processing selection unit reads from thezone dividing information storage unit a speed zone to which a speed ofthe mobile object detected by the moving information detection unitbelongs, reads from the zone dividing information storage unit adistance zone to which a distance to the object the image of which is tobe picked calculated by the distance calculation unit belongs, andselects an image processing method based on a combination of the readspeed zone and the read distance zone.
 13. The image processingapparatus according to claim 12, further comprising: a closed regionsetting unit which sets a closed region different for each of theplurality of distance zones; and an object detection unit which detectsa predetermined object for each closed region set by the closed regionsetting unit.
 14. The image processing apparatus according to claim 12,further comprising a selection method changing unit which changes amethod for selecting the image processing method in the processingselection unit.
 15. The image processing apparatus according to claim 1,wherein the imaging unit includes a pair of image pickup opticalsystems; and a pair of image pickup elements which convert an opticalsignal output from the pair of image pickup optical systems into anelectric signal.
 16. The image processing apparatus according to claim1, wherein the imaging unit includes a pair of waveguide optical system;and an image pickup element which has image pickup regions correspondingrespectively to the waveguide optical systems, and converts an opticalsignal guided by each waveguide optical system into an electric signalin the corresponding image pickup region.
 17. The image processingapparatus according to claim 1, wherein the mobile object is a vehicle.18. An image processing method comprising: picking an image of apredetermined field of view from a mobile object to generate an image;detecting moving information including a speed of the mobile object;setting contents of a process to be performed in the generated image,based on the detected moving information; and performing processingcalculation according to the contents of the process set.
 19. The imageprocessing method according to claim 18, further comprising: predictinga point at which the mobile object arrives a predetermined period oftime after based on the detected moving information, wherein the settingincludes setting a calculation range in which a distance to an objectincluded in the field of view based on a prediction result by thepredicting, and the performing processing calculation includesperforming distance calculation in the calculation range set.
 20. Theimage processing method according to claim 18, wherein the settingincludes selecting an image processing range to be calculated, from aplurality of image processing ranges included in the image, based on thedetected moving information, and switching to the selected imageprocessing range, and the performing processing calculation includescalculating a distance to an object included in the field of view basedon an image signal group corresponding to the switched image processingrange.
 21. The image processing method according to claim 18, furthercomprising: calculating a distance from an image pickup position in theimaging step to an object the image of which is to be picked, whereinthe setting includes selecting an image processing method from aplurality of image processing methods, based on the distance from theimage pickup position to the object the image of which is to be pickedand a speed of the mobile object, and the performing processingcalculation includes applying the selected image processing method toperform image processing to the image.
 22. A computer program producthaving a computer readable medium including programmed instructions forperforming image processing on an image generated by an imaging unitwhich is mounted on a mobile object and picks an image of apredetermined field of view to generate the image, wherein theinstructions, when executed by a computer, cause the computer toperform: detecting moving information including a speed of the mobileobject; setting contents of a process to be performed in the generatedimage, based on the detected moving information; and performingprocessing calculation according to the contents of the process set. 23.The computer program product according to claim 22, wherein theinstructions further cause the computer to perform predicting a point atwhich the mobile object arrives a predetermined period of time afterbased on the detected moving information, wherein the setting includessetting a calculation range in which a distance to an object included inthe field of view based on a prediction result by the predicting, andthe performing processing calculation includes performing distancecalculation in the calculation range set.
 24. The computer programproduct according to claim 22, wherein the setting includes selecting animage processing range to be calculated, from a plurality of imageprocessing ranges included in the image, based on the detected movinginformation, and switching to the selected image processing range, andthe performing processing calculation includes calculating a distance toan object included in the field of view based on an image signal groupcorresponding to the switched image processing range.
 25. The computerprogram product according to claim 22, wherein the instructions furthercause the computer to perform calculating a distance from an imagepickup position in the imaging step to an object the image of which isto be picked, wherein the setting includes selecting an image processingmethod from a plurality of image processing methods, based on thedistance from the image pickup position to the object the image of whichis to be picked and a speed of the mobile object, and the performingprocessing calculation includes applying the selected image processingmethod to perform image processing to the image.